LAB project

Resilient, low power and encrypted LAB CLOUD project.

Diagram

Data security

• Availability is via LAB project#HA & RSYNC & another location & another internet provider & public DNS
• Integrity is provided by BTRFS and will be provided by MINIO for large static files
• Theft protection is provided by AC and FDE encryption
• Loss protection is performed via AC & RSYNC & BTRFS and locks/unlocks FDE drives
• Scalability will be via the CEPH

My LAB tools

• MEDIAWIKI (sharing editor)
• DEBIAN
• app.diagrams.net (diagram editors)
• x86 CPU (for ARM I will use QEMU emulator)
• OPENCL GPU (AI&SIGNAL processing)
• LXD/SNAP/MICROK8S (virtualization&container)
• LSOF (system&network audit)
• NMAP (Network scanner)
• TCPDUMP (Network monitoring)

Tested in this LAB

• NEXTCLOUD nextcloud.infocepo.com (aFM8yKYmDa)
• MEDIAWIKI infocepo.com
• KUBERNETES Cluster wiki.infocepo.com (demo available, ask me)
• MARIADB GALERA Cluster
• OPENSTACK
• PROMETHEUS prometheus.infocepo.com (demo available, ask me)
• YACY google.infocepo.com (decentralized search engine) (demo available, ask me)
• GLUSTERFS
• KATA (container runtime)
• AWS CLI
• ANSIBLE
• GIT

HA

I love COROSYNC/PACEMAKER, but for this LAB I wanted something from scratch:

#!/bin/bash
#ynotopec at gmail.com

domainName=$(cat domainName)
portNumber=$(cat portNumber)
publicHost1=$(cat publicHost1)
publicHost2=$(cat publicHost2)

echo "$(date): init"

#Clean
stop${domainName}.sh

#If I was the master sleep a little !
ipPublic="$(dig +short myip.opendns.com @resolver1.opendns.com )"
ipMasterOld="$(nslookup ${domainName} |sed -rn 's#^Address: (.*)$#\1#p' )"
[ "${ipPublic}" = "${ipMasterOld}" ] &&sleep 240

#Wait Internet
while (! ping -w2 -c1 ${publicHost1} >/dev/null 2>&1 );do
  sleep 10
done

#Wait Admin Unlock Backup
waitAdminUnLockBCK.sh

#If passif merge backup
if (nc -zw2 ${domainName}  443 >/dev/null 2>&1 ||(sleep 10 ;nc -zw4 ${domainName}  443 >/dev/null 2>&1 ) ) ;then
  rsync --max-size=4M --ignore-existing --numeric-ids --modify-window=1 --ignore-errors --block-size=128.00K --inplace --no-whole-file \
   -z --compress-level=9 \
   -aAXxv \
   --exclude=.rsync_* \
   --rsh="ssh -i ~/.ssh/storage@${domainName}.key -p ${portNumer} -oStrictHostKeyChecking=no" \
   ${domainName}:/storage/rsync-rollback/ /storage/rsync-rollback/
fi

#lock bck source after merge
lockBckSource.sh

#Wait master down (big loop)
while (nc -zw2 ${domainName} ${portNumber} >/dev/null 2>&1 ||(sleep 10 ;nc -zw4 ${domainName} ${portNumber} >/dev/null 2>&1 ) );do
  if [ -z "$(find ha_synced -type f -mmin -60 2>/dev/null )" ] ;then
    rsync --numeric-ids --delete --force --modify-window=1 --ignore-errors --block-size=128.00K --inplace --no-whole-file \
     -z --compress-level=9 \
     -aAXxv \
     --backup-dir=rsync-rollback/$(date '+%Y-%m-%d') \
     --exclude=rsync-rollback \
     --exclude=.rsync_* \
     --rsh="ssh -i ~/.ssh/storage@${domainName}.key -p ${portNumer} -oStrictHostKeyChecking=no" \
     ${domainName}:/storage/ /storage/ &&touch ha_synced
  fi
  sleep 10
done

#Maybe it's me down ! If INTERNET down, reboot
! ping -w2 -c1 ${publicHost1} >/dev/null 2>&1 &&reboot

#Become master

#lock Backup (integrity protection against attacks)
lockBCK.sh

#Register DNS
ipMasterDown="$(nslookup ${domainName} |sed -rn 's#^Address: (.*)$#\1#p' )"
updateDns.sh "${ipPublic}"

#Random startup time
sleep $((RANDOM%10))
${domainName}Start.sh &

#Wait DNS propagation
sleep $((240-25))

#Monitor
echo "$(date): up"
[ "${ipPublic}" = "${ipMasterDown}" ] &&ipMasterDown=""
while [ "${ipPublic}" = "$(nslookup ${domainName} |sed -rn 's#^Address: (.*)$#\1#p' )" ] \
 &&( ! nc -zw2 "${ipMasterDown}" 443 >/dev/null 2>&1 ) \
 &&(nc -zw2 "${ipPublic}" 443 >/dev/null 2>&1 ||nc -zw4 "${ipPublic}" 443 >/dev/null 2>&1 ) \
 &&(nc -zw2 "${ipMasterDown}" ${portNumer} >/dev/null 2>&1 ||ping -w2 -c1 ${publicHost1} >/dev/null 2>&1 ||ping -w2 -c1 ${publicHost2} >/dev/null 2>&1 ) ;do
  sleep 4
done
echo "$(date): down"

#Stop
stop${domainName}.sh

reboot

Optimizations explained

To reduce Network data I added options to rsync:

• only send modified data blocks from files

--inplace --block-size=128.00K

• compress transfer

-z --compress-level=9

To increase flash storage life:

--inplace --no-whole-file

Cost per month

800€*2=1600€ :hypervisors (popular CPU X86 notebook, 8GB+)
-400€*2=-800€ :sale after one year
800€/12m=33€ :price/month
30*2=60€ :Internet (~1PB outbound/month)
(7.5W+7.5W)*24h*30.5d/1000Wh*.15€*2~=3.3€ :electricity

33+60+3.3~=96€ :cost/month

Futur

• Migrate rsync to real time replication.
• Add MINIO for objects storage and streaming.
• Improve synchronization algorithm (bisync.sh):

1. The NEXTCLOUD algorithm is very good average, but is very poor for dynamic big files like DB or virtual images.
2. OSYNC is slow, and I don't know for big files.
3. RSYNC is very good average but not dealing with inodes (moving files).

• Alert passive location locked and add de-lock page.
• Double storage to have active/active locations. Storage 1 location A to B. Storage 2 location B to A.