LAB project

Resilient, low power and encrypted LAB CLOUD project.

Diagram

Data security

• availability is done via LAB project#HA & RSYNC & another location & another Internet provider & public DNS
• integrity is done via BTRFS
• theft protection is done via AC & FDE encryption
• loss protection is done via AC & RSYNC & BTRFS & lock/unlock FDE disks
• scalability is be done via OPENSTACK

My LAB tools

• MEDIAWIKI (sharing editor)
• DEBIAN
• DIA & app.diagrams.net (diagram editors)
• x86 CPU (for ARM I will use QEMU emulator)
• OpenCL GPU (AI&SIGNAL processing)
• LXD/SNAP (virtualization&container)
• LSOF (system&network audit)
• nmap (Network scanner)
• tcpdump (Network monitoring)

Tested in this LAB

• NEXTCLOUD nextcloud.infocepo.com
• MEDIAWIKI infocepo.com
• KUBERNETES Cluster
• MARIADB GALERA Cluster
• OPENSTACK
• PROMETHEUS prometheus.infocepo.com (disabled)
• YACY (google.infocepo.com decentralized search engine) (disabled)
• GLUSTERFS
• KATA
• AWS CLI
• ANSIBLE
• GIT

HA

I love COROSYNC/PACEMAKER, but for this LAB I wanted something from scratch:

#!/bin/bash
#pacheco at infocepo.com

domainName=$(cat domainName)
portNumber=$(cat portNumber)
publicHost1=$(cat publicHost1)
publicHost2=$(cat publicHost2)

echo "$(date): init"

#Clean
stop${domainName}.sh

#If I was the master sleep a little !
ipPublic="$(dig +short myip.opendns.com @resolver1.opendns.com )"
ipMasterOld="$(nslookup ${domainName} |sed -rn 's#^Address: (.*)$#\1#p' )"
[ "${ipPublic}" = "${ipMasterOld}" ] &&sleep 240

#Wait master down (big loop)
while (nc -zw2 ${domainName} ${portNumber} >/dev/null 2>&1 ||(sleep 10 ;nc -zw4 ${domainName} ${portNumber} >/dev/null 2>&1 ) );do
  if [ -z "$(find ha_synced -type f -mmin -60 2>/dev/null )" ] ;then
    rsync --numeric-ids --delete --force --modify-window=1 --ignore-errors --block-size=128.00K --inplace --no-whole-file \
     -z --compress-level=9 \
     -aAXxv \
     --backup-dir=rsync-rollback/$(date '+%Y-%m-%d') \
     --exclude=rsync-rollback \
     --exclude=.rsync_* \
     --rsh="ssh -i ~/.ssh/storage@${domainName}.key -p ${portNumer} -oStrictHostKeyChecking=no" \
     ${domainName}:/storage/ /storage/ &&touch ha_synced
  fi
  sleep 10
done

#Maybe it's me down ! If INTERNET down, reboot
! ping -w2 -c1 ${publicHost} >/dev/null 2>&1 &&systemctl start kexec.target

#Become master

#Register DNS
ipMasterDown="$(nslookup ${domainName} |sed -rn 's#^Address: (.*)$#\1#p' )"
updateDns.sh "${ipPublic}"

#Random startup time
sleep $((RANDOM%10))
${domainName}Start.sh &

#Wait DNS propagation
sleep $((240-25))

#Monitor
echo "$(date): up"
[ "${ipPublic}" = "${ipMasterDown}" ] &&ipMasterDown=""
while [ "${ipPublic}" = "$(nslookup ${domainName} |sed -rn 's#^Address: (.*)$#\1#p' )" ] \
 &&( ! nc -zw2 "${ipMasterDown}" 443 >/dev/null 2>&1 ) \
 &&(nc -zw2 "${ipPublic}" 443 >/dev/null 2>&1 ||nc -zw4 "${ipPublic}" 443 >/dev/null 2>&1 ) \
 &&(nc -zw2 "${ipMasterDown}" ${portNumer} >/dev/null 2>&1 ||ping -w2 -c1 ${publicHost1} >/dev/null 2>&1 ||ping -w2 -c1 ${publicHost2} >/dev/null 2>&1 ) ;do
  sleep 4
done
echo "$(date): down"

#Stop
stop${domainName}.sh

#Reboot
systemctl start kexec.target

Optimizations explained

To reduce Network data I add options to rsync:

• only send modified data blocks from files

--inplace --block-size=128.00K

• compress transfert

-z --compress-level=9"

To increase flash storage life:

--inplace --no-whole-file

Futur

• Migrate rsync to real time replication.
• Add OPENSTACK SWIFT for objects storage and streaming
• Improve synchronization algorithm (bisync.sh):

-The NEXTCLOUD algorithm is very good average, but is very poor for dynamic big files like DB or virtual images. -OSYNC is slow, and I don't know for big files. -RSYNC is very good average but not dealing with inodes (moving files).

Other

If you have ideas to improve something, send me an e-mail (pacheco at infocepo.com).