LAB project

Resilient, low power and encrypted LAB CLOUD project.

Diagram

Data security

• availability is done via LAB project#HA & RSYNC & another location & another Internet provider & public DNS
• integrity is done via BTRFS & will be done by OPENSTACK SWIFT for static data
• theft protection is done via AC & FDE encryption
• loss protection is done via AC & RSYNC & BTRFS & lock/unlock FDE disks
• scalability can be done via OPENSTACK

My LAB tools

• MEDIAWIKI (sharing editor)
• DEBIAN
• DIA & app.diagrams.net (diagram editors)
• x86 CPU (for ARM I will use QEMU emulator)
• OpenCL GPU (AI&SIGNAL processing)
• LXD/SNAP/KUBERNETES (virtualization&container)
• LSOF (system&network audit)
• nmap (Network scanner)
• tcpdump (Network monitoring)

Tested in this LAB

• NEXTCLOUD nextcloud.infocepo.com
• MEDIAWIKI infocepo.com
• KUBERNETES Cluster wiki.infocepo.com (demo available, ask me to start)
• MARIADB GALERA Cluster
• OPENSTACK
• PROMETHEUS prometheus.infocepo.com (demo available, ask me to start)
• YACY google.infocepo.com (decentralized search engine) (demo available, ask me to start)
• GLUSTERFS
• KATA
• AWS CLI
• ANSIBLE
• GIT

HA

I love COROSYNC/PACEMAKER, but for this LAB I wanted something from scratch:

#!/bin/bash
#pacheco at infocepo.com

domainName=$(cat domainName)
portNumber=$(cat portNumber)
publicHost1=$(cat publicHost1)
publicHost2=$(cat publicHost2)

echo "$(date): init"

#Clean
stop${domainName}.sh

#If I was the master sleep a little !
ipPublic="$(dig +short myip.opendns.com @resolver1.opendns.com )"
ipMasterOld="$(nslookup ${domainName} |sed -rn 's#^Address: (.*)$#\1#p' )"
[ "${ipPublic}" = "${ipMasterOld}" ] &&sleep 240

#Wait Internet
while (! ping -w2 -c1 ${publicHost1} >/dev/null 2>&1 );do
  sleep 10
done

#Wait master down (big loop)
while (nc -zw2 ${domainName} ${portNumber} >/dev/null 2>&1 ||(sleep 10 ;nc -zw4 ${domainName} ${portNumber} >/dev/null 2>&1 ) );do
  if [ -z "$(find ha_synced -type f -mmin -60 2>/dev/null )" ] ;then
    rsync --numeric-ids --delete --force --modify-window=1 --ignore-errors --block-size=128.00K --inplace --no-whole-file \
     -z --compress-level=9 \
     -aAXxv \
     --backup-dir=rsync-rollback/$(date '+%Y-%m-%d') \
     --exclude=rsync-rollback \
     --exclude=.rsync_* \
     --rsh="ssh -i ~/.ssh/storage@${domainName}.key -p ${portNumer} -oStrictHostKeyChecking=no" \
     ${domainName}:/storage/ /storage/ &&touch ha_synced
  fi
  sleep 10
done

#Maybe it's me down ! If INTERNET down, reboot
! ping -w2 -c1 ${publicHost1} >/dev/null 2>&1 &&systemctl start kexec.target

#Become master

#Register DNS
ipMasterDown="$(nslookup ${domainName} |sed -rn 's#^Address: (.*)$#\1#p' )"
updateDns.sh "${ipPublic}"

#Random startup time
sleep $((RANDOM%10))
${domainName}Start.sh &

#Wait DNS propagation
sleep $((240-25))

#Monitor
echo "$(date): up"
[ "${ipPublic}" = "${ipMasterDown}" ] &&ipMasterDown=""
while [ "${ipPublic}" = "$(nslookup ${domainName} |sed -rn 's#^Address: (.*)$#\1#p' )" ] \
 &&( ! nc -zw2 "${ipMasterDown}" 443 >/dev/null 2>&1 ) \
 &&(nc -zw2 "${ipPublic}" 443 >/dev/null 2>&1 ||nc -zw4 "${ipPublic}" 443 >/dev/null 2>&1 ) \
 &&(nc -zw2 "${ipMasterDown}" ${portNumer} >/dev/null 2>&1 ||ping -w2 -c1 ${publicHost1} >/dev/null 2>&1 ||ping -w2 -c1 ${publicHost2} >/dev/null 2>&1 ) ;do
  sleep 4
done
echo "$(date): down"

#Stop
stop${domainName}.sh

#Reboot
systemctl start kexec.target

Optimizations explained

To reduce Network data I added options to rsync:

• only send modified data blocks from files

--inplace --block-size=128.00K

• compress transfer

-z --compress-level=9"

To increase flash storage life:

--inplace --no-whole-file

Cost per month

800€*2=1600€ :hypervisors (popular CPU X86 notebook, 8GB+)
-400€*2=-800€ :sale after one year
800€/12m=33€ :price/month
30*2=60€ :Internet (~1PB outbound/month)
(7.5W+7.5W)*24h*30.5d/1000Wh*.15€*2~=3.3€ :electricity

33+60+3.3~=96€ :cost/month

Futur

• Migrate rsync to real time replication.
• Add OPENSTACK SWIFT for objects storage and streaming.
• Improve synchronization algorithm (bisync.sh):

1. The NEXTCLOUD algorithm is very good average, but is very poor for dynamic big files like DB or virtual images.
2. OSYNC is slow, and I don't know for big files.
3. RSYNC is very good average but not dealing with inodes (moving files).

• Alert passive location locked and add de-lock page.