Lab notes

OpenVPN endpoint

The lab has a virtual machine for NAT, which acts as a bridge between the internet (WAN) and the private networks within the AWS Cloud (VPCs).

(labmgr) marius@mleng:~/lab-manager/home_pub/student/Desktop/lab-manager$ aws ec2 describe-instances --query "Reservations[*].Instances[*].[Tags[?Key=='Name'].Value | [0], InstanceId, PublicIpAddress, PrivateIpAddress]" --output table
----------------------------------------------------------------------------
|                             DescribeInstances                            |
+--------------+-----------------------+----------------+------------------+
|  DC2         |  i-06b4cfdc9d7294e11  |  None          |  192.168.20.103  |
|  C2          |  i-06022fc3b129f04d6  |  None          |  192.168.20.107  |
|  SQL         |  i-057b58cc1bdc58935  |  None          |  192.168.20.104  |
|  DC          |  i-0ac391b51cfd3fec2  |  None          |  192.168.20.101  |
|  WIN19       |  i-0b00a16f81ef2857f  |  None          |  192.168.20.102  |
|  WIN10       |  i-0c05863d51b58aa1d  |  None          |  192.168.20.105  |
|  SOC         |  i-092fa119ac01ed955  |  None          |  192.168.20.106  |
|  CommandoVM  |  i-016d72bd9b3ca098f  |  54.78.96.249  |  192.168.0.200   |
|  NAT instance|  i-09534d4f486c4191f  |  3.249.213.213 |  192.168.0.62    |
+--------------+-----------------------+----------------+------------------+

NAT instance is the VPN endpoint. The external IPs are dynamic.
labmgr is a prompt indicator, for a virtualenv (Python)

OpenVPN route for Elasticsearch DB access

The .ovpn config needs to be extended for the route:

...
-----END OpenVPN Static key V1-----
</tls-auth>


# Add this line to route traffic to 192.168.20.106 through the VPN
route 192.168.20.106 255.255.255.255

GitHub file site limit

100 MB for repository files
2000 MB for release files

Files can be compressed and split, with the command line utilities.

Due to these limitations, the data cannot get pushed directly into the repo. The release function is used instead. When necessary, files gets split.

remote: error: File data/lab_logs_normal_activity_may_6_2024.json is 1811.83 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
To https://github.com/norandom/log2ml.git
 ! [remote rejected] main -> main (pre-receive hook declined)
error: failed to push some refs to 'https://github.com/norandom/log2ml.git

Download of release data with Python

Downloads of release assets are difficult because GitHub requires authenticated links (API), and has multiple redirects. The download functions for released data have been implemented in a separate library, which handles such tasks.