I am mainly used to the enterprise where most connections are deployed by some "normal" kind of fiber eg SFP+, QSFP+, SFP28, QSFP28 and those modules are almost fool proof, run cool and are reliable.  Like most normal admin users, I like to use my own hardware for routing and switching rather than the often poor equipment provided by the ISP, especially when you have enterprise hardware that will be more reliable.

The normal path is that you would just take the fiber optic link on an SFP or SFP+ and plug it in directly into your firewall, router or switch, cutting out the extra power usage, insecure ISP devices and another point of failure.  

But enter the situation with a lot of home and office connections and anything on ISPs like Telus, AT&T, most ISPs in Japan etc... that are faster than 1.5G, they no longer use the standard GPON which just registers via the serial# of the module that the ISP has on file.

Now you need a device called the NAH (Network Access Hub) which uses XGSPON which is authenticated via this extra and unnecessary device/NAH.  Plugging in an XGSPON SFP+ module to your router/switch will not work like a normal GPON.

Solution 1 - buy an expensive XGSPON module and hack and modify it

After more testing, I almost think that #1 in some cases here may be more reliable, yes there is the chance of authentication parameters changing which would instantly take you offline.  However, many of the SFP+ to RJ45 adapters are known to be unreliable and randomly die.  In theory if your XGSPON module doesn't die, then I will predict it will probably be more reliable than the #2 adapter method.

There are some solutions which modify XGSPON modules from fs.com so you can use it as normal, but they are pricey and this can be risky if the authentication mode/details changes in the future.  In plain English, you could be away from home and need access to your home network but an ISP Update could render your internet access useless.  Here is the procedure on Github to hack the XGSPON SFP+ module from fs.com.

Solution 2 - Use the NAH and use an SFP+ to RJ45 module

As much as I hate the NAH, this is the most reliable way for the reasons mentioned in #1.  With a caveat being that I don't trust these adapters, they can die.  For example one of mine died after 3 days.  They do say the 80M are more reliable.

Recommended 80M SFP+ to RJ45

The 80M or 100M units use the BCM84891L Broadcom chip which is considered more reliable in my experience.

My 80M unit has been going strong for over a month now and temps vary between 42-48C which appears to be based on usage, whereas the 30M units seem to stay at almost a static temperature.

Amazon affil Link to the 80M 10G SFP+ RJ45 adapter I use.

I don't recommend the 30M SFP+ modules anymore below. 

The 30Ms almost always use the Marvell AQR113C as mentioned below which is a chip that is proven to overheat and not last (eg. mine died after just 3 days!).

One module I recently tested has an ID of "SFP+-10G-SR" and runs at a reasonable 51C which is much better than what many users report with other modules (eg. burning hot 80-90C).  I cannot be certain if the unit runs cool because of the design it claims to use which minimizes heat, or if it is because my enterprise switch has active cooling.

It seems almost all 30M SFP+ modules use the Marvell AQR113C chip which no matter the brand is often not reliable.

The 80M or 100M SFP+ modules usually use Broadcom BCM84891L which is known to be a cooler running chip.  It does cost more but if they last longer then it is worth it.

After testing with heavy use for 3-4 days the temperature stayed the same but the module dies/drops out which causes downtime/packetloss:

Jan 25 13:04:08   chassism[1243]:  link 3 SFP receive power low  warning cleared
Jan 25 13:07:31   mib2d[1262]: SNMP_TRAP_LINK_DOWN: ifIndex 561, ifAdminStatus up(1), ifOperStatus down(2), ifName xe-0/1/3
Jan 25 13:07:33   chassism[1243]:  link 3 SFP receive power low  alarm set
Jan 25 13:07:33   chassism[1243]:  link 3 SFP receive power low  warning set
Jan 25 13:07:43   chassism[1243]:  link 3 SFP receive power low  alarm cleared
Jan 25 13:07:43   chassism[1243]:  link 3 SFP receive power low  warning cleared

Here is the affiliate link to the bad/dying after 3 days SFP+ to RJ45 item I bought.

Even when pushing 3Gigabit in both directions the module stayed at the same temp or maybe went to 52C.

These are fairly classic symptoms but they may not present obviously enough early on.

I had a power supply issue with my old Corsair CX430 (430 Watt).

The build had two old 95W Xeon CPUs, 2 USB devices and a PCI-E powered Nvidia Quadro or at one point a GT710.  This shouldn't have overwhelmed the PS but this power supply is known to be weak by many users.

Symptom 1. Graphics Issues

This would only present it self after coming back with the screensaver, when hitting the keyboard or mouse to stop the screensaver and get the login screen, it would just freeze for a number of seconds.  I initially chalked it up to bad cooling or an old Nvidia driver.

Symptom 2. Mouse Issues

Sometimes the mouse would just freeze or not track properly, clearly in hindsight it was a power supply issue but the mouse itself also did test bad, so this was not completely obvious.

Symptom 3. USB Issues

I didn't have this particular issue but a lot of times I've seen issues with power supplies, you may notice a USB SSD does not work right or drops out.  This is tricky to diagnose as it is often the USB SSD enclosure/cable itself, OR it is sometimes the internal cabling or resistance from motherboard to the front of the PC.

Symptom 4. Resetting randomly.

This one is fairly obvious but may slowly creep up.  You may assume if you aren't at the PC all the time that it may be due to an update or even a bad motherboard or a watchdog service acting up.

Conclusion

If you ever have any issues like this, if practical, swap out for a known good power supply and see if any of these types of symptoms goes away.

In my example, I swapped in a known good 750W EVGA and all has been fine since (no resets, no screensaver freezing, no USB issues etc...)

In this quick tutorial we will deploy various AI models and compare the differences.  We will be using smaller models so that most users should be able to follow this guide as long as they have at least 16G of memory and no other applications using much memory.

In this tutorial we recommend that you use Docker to deploy ollama.  If you are not familiar with Docker, you can check our Docker Tutorial guide here.

Step 1.) Deploy

I recommend having a Ubuntu or Debian host for this, but it is supported in Windows and Mac too.

Visit the Ollama website to download it.

Step 2.) Install Ollama

Install curl if you don't have it already: apt update && apt install curl

curl -fsSL https://ollama.com/install.sh | sh

 >>> Installing ollama to /usr/local
>>> Downloading Linux amd64 bundle
######################################################################## 100.0%
WARNING: Unable to detect NVIDIA/AMD GPU. Install lspci or lshw to automatically detect and install GPU dependencies.
>>> The Ollama API is now available at 127.0.0.1:11434.
>>> Install complete. Run "ollama" from the command line.


Step 3.) Start ollama server

ollama serve&

Make sure you use the & so it puts the service in the background, yet it will still output to the console.  This is very useful for debugging and troubleshooting performance issues.


2025/04/22 21:00:21 routes.go:1231: INFO server config env="map[CUDA_VISIBLE_DEVICES: GPU_DEVICE_ORDINAL: HIP_VISIBLE_DEVICES: HSA_OVERRIDE_GFX_VERSION: HTTPS_PROXY: HTTP_PROXY: NO_PROXY: OLLAMA_CONTEXT_LENGTH:2048 OLLAMA_DEBUG:false OLLAMA_FLASH_ATTENTION:false OLLAMA_GPU_OVERHEAD:0 OLLAMA_HOST:http://127.0.0.1:11434 OLLAMA_INTEL_GPU:false OLLAMA_KEEP_ALIVE:5m0s OLLAMA_KV_CACHE_TYPE: OLLAMA_LLM_LIBRARY: OLLAMA_LOAD_TIMEOUT:5m0s OLLAMA_MAX_LOADED_MODELS:0 OLLAMA_MAX_QUEUE:512 OLLAMA_MODELS:/root/.ollama/models OLLAMA_MULTIUSER_CACHE:false OLLAMA_NEW_ENGINE:false OLLAMA_NOHISTORY:false OLLAMA_NOPRUNE:false OLLAMA_NUM_PARALLEL:0 OLLAMA_ORIGINS:[http://localhost https://localhost http://localhost:* https://localhost:* http://127.0.0.1 https://127.0.0.1 http://127.0.0.1:* https://127.0.0.1:* http://0.0.0.0 https://0.0.0.0 http://0.0.0.0:* https://0.0.0.0:* app://* file://* tauri://* vscode-webview://* vscode-file://*] OLLAMA_SCHED_SPREAD:false ROCR_VISIBLE_DEVICES: http_proxy: https_proxy: no_proxy:]"
time=2025-04-22T21:00:21.697Z level=INFO source=images.go:458 msg="total blobs: 0"
time=2025-04-22T21:00:21.697Z level=INFO source=images.go:465 msg="total unused blobs removed: 0"
time=2025-04-22T21:00:21.697Z level=INFO source=routes.go:1298 msg="Listening on 127.0.0.1:11434 (version 0.6.5)"
time=2025-04-22T21:00:21.697Z level=INFO source=gpu.go:217 msg="looking for compatible GPUs"
time=2025-04-22T21:00:21.727Z level=INFO source=gpu.go:377 msg="no compatible GPUs were discovered"
time=2025-04-22T21:00:21.727Z level=INFO source=types.go:130 msg="inference compute" id=0 library=cpu variant="" compute="" driver=0.0 name="" total="267.6 GiB" available="255.5 GiB"
 

Step 4.) Run your first llm!

Find the LLM you want to test on the ollama website.

Keep in mind that the larger the model, the more memory and more disk space and resources it consumes when running.  For this reason, our example will use a smaller model.

ollama run qwen2.5:0.5b

Note that this smaller 0.5b model uses just about 397MB

You should see similar output as below and then have a chat prompt at the end.

 ollama run qwen2.5:0.5b
[GIN] 2025/04/22 - 21:04:44 | 200 |      79.024µs |       127.0.0.1 | HEAD     "/"
[GIN] 2025/04/22 - 21:04:44 | 404 |     455.218µs |       127.0.0.1 | POST     "/api/show"
pulling manifest ⠇ time=2025-04-22T21:04:45.299Z level=INFO source=download.go:177 msg="downloading c5396e06af29 in 4 100 MB part(s)"
pulling manifest
pulling manifest
pulling c5396e06af29... 100% ▕██████████████████████████████████████████████████████████████████████████████▏ 397 MB                         tpulling manifest
pulling c5396e06af29... 100% ▕██████████████████████████████████████████████████████████████████████████████▏ 397 MB                         
pulling manifest
pulling c5396e06af29... 100% ▕██████████████████████████████████████████████████████████████████████████████▏ 397 MB                         
pulling manifest
pulling c5396e06af29... 100% ▕██████████████████████████████████████████████████████████████████████████████▏ 397 MB                         
pulling manifest
pulling manifest
pulling c5396e06af29... 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 397 MB                         
pulling 66b9ea09bd5b... 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏   68 B                         
pulling eb4402837c78... 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏ 1.5 KB                         
pulling 832dd9e00a68... 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏  11 KB                         
pulling 005f95c74751... 100% ▕██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▏  490 B                         
verifying sha256 digest
writing manifest
success
[GIN] 2025/04/22 - 21:04:57 | 200 |   66.099807ms |       127.0.0.1 | POST     "/api/show"
⠙ time=2025-04-22T21:04:57.759Z level=INFO source=server.go:105 msg="system memory" total="267.6 GiB" free="255.5 GiB" free_swap="976.0 MiB"
time=2025-04-22T21:04:57.759Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.vision.block_count default=0
time=2025-04-22T21:04:57.759Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.key_length default=64
time=2025-04-22T21:04:57.759Z level=WARN source=ggml.go:152 msg="key not found" key=qwen2.attention.value_length default=64
time=2025-04-22T21:04:57.759Z level=INFO source=server.go:138 msg=offload library=cpu layers.requested=-1 layers.model=25 layers.offload=0 layers.split="" memory.available="[255.5 GiB]" memory.gpu_overhead="0 B" memory.required.full="782.6 MiB" memory.required.partial="0 B" memory.required.kv="96.0 MiB" memory.required.allocations="[782.6 MiB]" memory.weights.total="373.7 MiB" memory.weights.repeating="235.8 MiB" memory.weights.nonrepeating="137.9 MiB" memory.graph.full="298.5 MiB" memory.graph.partial="405.0 MiB"
llama_model_loader: loaded meta data with 34 key-value pairs and 290 tensors from /root/.ollama/models/blobs/sha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc870eda801d3ab0515 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 0.5B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Qwen2.5
llama_model_loader: - kv   5:                         general.size_label str              = 0.5B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                       general.license.link str              = https://huggingface.co/Qwen/Qwen2.5-0...
llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
llama_model_loader: - kv   9:                  general.base_model.0.name str              = Qwen2.5 0.5B
llama_model_loader: - kv  10:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  11:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2.5-0.5B
llama_model_loader: - kv  12:                               general.tags arr[str,2]       = ["chat", "text-generation"]
llama_model_loader: - kv  13:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  14:                          qwen2.block_count u32              = 24
llama_model_loader: - kv  15:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv  16:                     qwen2.embedding_length u32              = 896
llama_model_loader: - kv  17:                  qwen2.feed_forward_length u32              = 4864
llama_model_loader: - kv  18:                 qwen2.attention.head_count u32              = 14
llama_model_loader: - kv  19:              qwen2.attention.head_count_kv u32              = 2
llama_model_loader: - kv  20:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  21:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  22:                          general.file_type u32              = 15
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = qwen2
⠹ llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", """, "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  30:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {%- if tools %}n    {{- '<|im_start|>...
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  121 tensors
llama_model_loader: - type q5_0:  132 tensors
llama_model_loader: - type q8_0:   13 tensors
llama_model_loader: - type q4_K:   12 tensors
llama_model_loader: - type q6_K:   12 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 373.71 MiB (6.35 BPW)
⠼ load: special tokens cache size = 22
⠴ load: token to piece cache size = 0.9310 MB
print_info: arch             = qwen2
print_info: vocab_only       = 1
print_info: model type       = ?B
print_info: model params     = 494.03 M
print_info: general.name     = Qwen2.5 0.5B Instruct
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
llama_model_load: vocab only - skipping tensors
time=2025-04-22T21:04:58.219Z level=INFO source=server.go:405 msg="starting llama server" cmd="/usr/local/bin/ollama runner --model /root/.ollama/models/blobs/sha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc870eda801d3ab0515 --ctx-size 8192 --batch-size 512 --threads 12 --no-mmap --parallel 4 --port 38569"
time=2025-04-22T21:04:58.219Z level=INFO source=sched.go:451 msg="loaded runners" count=1
time=2025-04-22T21:04:58.219Z level=INFO source=server.go:580 msg="waiting for llama runner to start responding"
time=2025-04-22T21:04:58.220Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server error"
time=2025-04-22T21:04:58.241Z level=INFO source=runner.go:853 msg="starting go runner"
time=2025-04-22T21:04:58.243Z level=INFO source=ggml.go:109 msg=system CPU.0.LLAMAFILE=1 compiler=cgo(gcc)
time=2025-04-22T21:04:58.250Z level=INFO source=runner.go:913 msg="Server listening on 127.0.0.1:38569"
⠦ llama_model_loader: loaded meta data with 34 key-value pairs and 290 tensors from /root/.ollama/models/blobs/sha256-c5396e06af294bd101b30dce59131a76d2b773e76950acc870eda801d3ab0515 (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 0.5B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Qwen2.5
llama_model_loader: - kv   5:                         general.size_label str              = 0.5B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                       general.license.link str              = https://huggingface.co/Qwen/Qwen2.5-0...
llama_model_loader: - kv   8:                   general.base_model.count u32              = 1
llama_model_loader: - kv   9:                  general.base_model.0.name str              = Qwen2.5 0.5B
llama_model_loader: - kv  10:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  11:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2.5-0.5B
llama_model_loader: - kv  12:                               general.tags arr[str,2]       = ["chat", "text-generation"]
llama_model_loader: - kv  13:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  14:                          qwen2.block_count u32              = 24
llama_model_loader: - kv  15:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv  16:                     qwen2.embedding_length u32              = 896
llama_model_loader: - kv  17:                  qwen2.feed_forward_length u32              = 4864
llama_model_loader: - kv  18:                 qwen2.attention.head_count u32              = 14
llama_model_loader: - kv  19:              qwen2.attention.head_count_kv u32              = 2
llama_model_loader: - kv  20:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  21:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  22:                          general.file_type u32              = 15
llama_model_loader: - kv  23:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  24:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  25:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", """, "#", "$", "%", "&", "'", ...
⠧ llama_model_loader: - kv  26:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  27:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  28:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  29:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  30:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  31:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  32:                    tokenizer.chat_template str              = {%- if tools %}n    {{- '<|im_start|>...
llama_model_loader: - kv  33:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  121 tensors
llama_model_loader: - type q5_0:  132 tensors
llama_model_loader: - type q8_0:   13 tensors
llama_model_loader: - type q4_K:   12 tensors
llama_model_loader: - type q6_K:   12 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q4_K - Medium
print_info: file size   = 373.71 MiB (6.35 BPW)
⠇ time=2025-04-22T21:04:58.472Z level=INFO source=server.go:614 msg="waiting for server to become available" status="llm server loading model"
⠏ load: special tokens cache size = 22
⠋ load: token to piece cache size = 0.9310 MB
print_info: arch             = qwen2
print_info: vocab_only       = 0
print_info: n_ctx_train      = 32768
print_info: n_embd           = 896
print_info: n_layer          = 24
print_info: n_head           = 14
print_info: n_head_kv        = 2
print_info: n_rot            = 64
print_info: n_swa            = 0
print_info: n_embd_head_k    = 64
print_info: n_embd_head_v    = 64
print_info: n_gqa            = 7
print_info: n_embd_k_gqa     = 128
print_info: n_embd_v_gqa     = 128
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: n_ff             = 4864
print_info: n_expert         = 0
print_info: n_expert_used    = 0
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 32768
print_info: rope_finetuned   = unknown
print_info: ssm_d_conv       = 0
print_info: ssm_d_inner      = 0
print_info: ssm_d_state      = 0
print_info: ssm_dt_rank      = 0
print_info: ssm_dt_b_c_rms   = 0
print_info: model type       = 1B
print_info: model params     = 494.03 M
print_info: general.name     = Qwen2.5 0.5B Instruct
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 151643 '<|endoftext|>'
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors:          CPU model buffer size =   373.71 MiB
⠹ llama_init_from_model: n_seq_max     = 4
llama_init_from_model: n_ctx         = 8192
llama_init_from_model: n_ctx_per_seq = 2048
llama_init_from_model: n_batch       = 2048
llama_init_from_model: n_ubatch      = 512
llama_init_from_model: flash_attn    = 0
llama_init_from_model: freq_base     = 1000000.0
llama_init_from_model: freq_scale    = 1
llama_init_from_model: n_ctx_per_seq (2048) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 8192, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 24, can_shift = 1
llama_kv_cache_init:        CPU KV buffer size =    96.00 MiB
llama_init_from_model: KV self size  =   96.00 MiB, K (f16):   48.00 MiB, V (f16):   48.00 MiB
llama_init_from_model:        CPU  output buffer size =     2.33 MiB
⠸ llama_init_from_model:        CPU compute buffer size =   300.25 MiB
llama_init_from_model: graph nodes  = 846
llama_init_from_model: graph splits = 1
time=2025-04-22T21:04:58.973Z level=INFO source=server.go:619 msg="llama runner started in 0.75 seconds"
[GIN] 2025/04/22 - 21:04:58 | 200 |  1.319178734s |       127.0.0.1 | POST     "/api/generate"
>>> Send a message (/? for help)

Checking the ollama process we can see it uses about

1589132 root      20   0 2874.7m 553.7m  21.5m S  1140   0.2   3:03.60 ollama

553.7MB of RAM

This is the official guide here from OpenEBS.

Step 1.) Make sure you have the right version.

As of this time 2025-04, we need Kubernetes/Microk8s version 1.23 or higher (note this will continue to increment higher).

Usually a good sign of having the wrong/old version is that you will encounter namespace and other errors:

error: unknown flag: --namespace

You can follow my quick install guide and make sure you manually update the version to whatever we need.

Step 2.) Enable Helm

microk8s enable helm

Step 3.) Get OpenEBS repo & update

microk8s helm repo add openebs https://openebs.github.io/openebs

microk8s helm repo update

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "openebs" chart repository
Update Complete. ⎈Happy Helming!⎈


Step 4.) Install OpenEBS

microk8s helm install openebs --namespace openebs openebs/openebs --create-namespace

NAME: openebs
LAST DEPLOYED: Thu Apr 17 17:38:46 2025
NAMESPACE: openebs
STATUS: deployed
REVISION: 1
NOTES:
Successfully installed OpenEBS.

Check the status by running: kubectl get pods -n openebs

The default values will install both Local PV and Replicated PV. However,
the Replicated PV will require additional configuration to be fuctional.
The Local PV offers non-replicated local storage using 3 different storage
backends i.e Hostpath, LVM and ZFS, while the Replicated PV provides one replicated highly-available
storage backend i.e Mayastor.

For more information,
- view the online documentation at https://openebs.io/docs
- connect with an active community on our Kubernetes slack channel.
        - Sign up to Kubernetes slack: https://slack.k8s.io
        - #openebs channel: https://kubernetes.slack.com/messages/openebs
 

03:00.0 RAID bus controller: Broadcom / LSI MegaRAID SAS 2208 [Thunderbolt] (rev 05)
 

This is risky but if you have an extra RAID/SAS card to replace it then you can try this to get IT mode.

https://docs.broadcom.com/docs-and-downloads/host-bus-adapters/host-bus-adapters-common-files/sas_sata_6g_p14/9207_8e_Package_P14_IT_Firmware_BIOS_for_MSDOS_Windows.zip

https://docs.broadcom.com/docs/12350477

Get the sas2flash for Linux:

https://docs.broadcom.com/docs-and-downloads/host-bus-adapters/host-bus-adapters-common-files/sas_sata_6g_p20/Installer_P20_for_Linux.zip

unzip Installer_P20_for_Linux.zip
Archive:  Installer_P20_for_Linux.zip
  inflating: Installer_P20_for_Linux/README_Installer_P20_Linux.txt  
   creating: Installer_P20_for_Linux/sas2flash_linux_i686_x86-64_rel/
  inflating: Installer_P20_for_Linux/sas2flash_linux_i686_x86-64_rel/sas2flash  
   creating: Installer_P20_for_Linux/sas2flash_linux_ppc64_rel/
  inflating: Installer_P20_for_Linux/sas2flash_linux_ppc64_rel/sas2flash  
  inflating: Installer_P20_for_Linux/SAS2FLASH_Phase20.0-20.00.00.00.pdf  
  inflating: Installer_P20_for_Linux/SAS2Flash_ReferenceGuide.pdf  
 

chmod +x Installer_P20_for_Linux/sas2flash_linux_i686_x86-64_rel/sas2flash
 

Save Your Card Details:

 Installer_P20_for_Linux/sas2flash_linux_i686_x86-64_rel/sas2flash -list -c 0
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    No LSI SAS adapters found! Limited Command Set Available!
    ERROR: Command Not allowed without an adapter!
    ERROR: Couldn't Create Command -list
    Exiting Program.
 

P14 sas2flash won't run on Ubuntu 20 or similar/newer:

sas2flash: sas2flash: cannot execute binary file

Usually when I come across a system like this, I just flash the firmware to an IT mode so we get just an AHCI system with JBOD, but sometimes it is not practical for remote systems or if we fear that flashing is too risky if something goes wrong.

In this case, we unfortunately have to use LSI/Broadcom proprietary CLI tools (megacli) to even make the drives visible.

Step 1 - Download MegaCLI

The official broadcom packages lack .deb packages and only have .rpm for Linux

At this time the latest MegaCLI 5.5 P2 can be downloaded from here.
 

Step 2 - Extract MegaCLI

unzip 8-07-14_MegaCLI.zip

Archive:  8-07-14_MegaCLI.zip
  inflating: 8.07.14_MegaCLI.txt     
  inflating: DOS/MegaCLI.exe         
 extracting: FreeBSD/MegaCLI.zip     
 extracting: FreeBSD/MegaCli64.zip   
  inflating: Linux/MegaCli-8.07.14-1.noarch.rpm  
  inflating: Solaris/MegaCli.pkg     
  inflating: Windows/MegaCli.exe     
  inflating: Windows/MegaCli64.exe 

Step 3 - Convert to .deb using alien

apt install libncurses5 alien

alien MegaCli-8.07.06-1.noarch.rpm
Warning: Skipping conversion of scripts in package MegaCli: postinst postrm
Warning: Use the --scripts parameter to include the scripts.
megacli_8.07.06-2_all.deb generated

Step 4 - Install .deb

dpkg -i megacli_8.07.06-2_all.deb
Selecting previously unselected package megacli.
(Reading database ... 295031 files and directories currently installed.)
Preparing to unpack megacli_8.07.06-2_all.deb ...
Unpacking megacli (8.07.06-2) ...
Setting up megacli (8.07.06-2) ...
Processing triggers for libc-bin (2.31-0ubuntu9) ...

/opt/MegaRAID/MegaCli/MegaCli64: error while loading shared libraries: libncurses.so.5: cannot open shared object file: No such file or directory

If you get the above: apt install libncurses5
 

Step 5 - Run megacli

root@mint:~/Linux# /opt/MegaRAID/MegaCli/MegaCli64 -h
                                     

      MegaCLI SAS RAID Management Tool  Ver 8.07.06 Nov 13, 2012

    (c)Copyright 2011, LSI Corporation, All Rights Reserved.


NOTE: The following options may be given at the end of any command below:

    [-Silent] [-AppLogFile filename] [-NoLog] [-page[N]]
                 [-] is optional.
                  N - Number of lines per page.

MegaCli -v
MegaCli -help|-h|?
MegaCli -adpCount
MegaCli -AdpSetProp {CacheFlushInterval -val} | { RebuildRate -val}
    | {PatrolReadRate -val} | {BgiRate -val} | {CCRate -val} | {ForceSGPIO -val}
    | {ReconRate -val} | {SpinupDriveCount -val} | {SpinupDelay -val}
    | {CoercionMode -val} | {ClusterEnable -val} | {PredFailPollInterval -val}
    | {BatWarnDsbl -val} | {EccBucketSize -val} | {EccBucketLeakRate -val}
    | {AbortCCOnError -val} | AlarmEnbl | AlarmDsbl | AlarmSilence
    | {SMARTCpyBkEnbl -val} | {SSDSMARTCpyBkEnbl -val} | NCQEnbl | NCQDsbl
    | {MaintainPdFailHistoryEnbl -val} | {RstrHotSpareOnInsert -val}
    | {DisableOCR -val} | {BootWithPinnedCache -val} | {enblPI -val} |{PreventPIImport -val}
    | AutoEnhancedImportEnbl | AutoEnhancedImportDsbl
    | {EnblSpinDownUnConfigDrvs -val}|{UseDiskActivityforLocate -val} -aN|-a0,1,2|-aALL
    | {ExposeEnclDevicesEnbl -val} | {SpinDownTime -val}
    | {SpinUpEncDrvCnt -val} | {SpinUpEncDelay -val} | {Perfmode -val} -aN|-a0,1,2|-aALL
    | {PerfMode -val �MaxFlushLines -val �MaxPDLatencyMS -val} -aN|-a0,1,2|-aALL
MegaCli -AdpSetProp -AutoDetectBackPlaneDsbl -val -aN|-a0,1,2|-aALL
       val - 0=Enable Auto Detect of SGPIO and i2c SEP.
             1=Disable Auto Detect of SGPIO.
             2=Disable Auto Detect of i2c SEP.
             3=Disable Auto Detect of SGPIO and i2c SEP.
MegaCli -AdpSetProp -CopyBackDsbl -val -aN|-a0,1,2|-aALL
       val - 0=Enable Copyback.
             1=Disable Copyback.
MegaCli -AdpSetProp -EnableJBOD -val -aN|-a0,1,2|-aALL
       val - 0=Disable JBOD mode.
             1=Enable JBOD mode.
MegaCli -AdpSetProp -DsblCacheBypass -val -aN|-a0,1,2|-aALL
       val - 0=Enable Cache Bypass.
             1=Disable Cache Bypass.
MegaCli -AdpSetProp -LoadBalanceMode -val -aN|-a0,1,2|-aALL
       val - 0=Auto Load balance mode.
             1=Disable Load balance mode.
MegaCli -AdpSetProp -UseFDEOnlyEncrypt -val -aN|-a0,1,2|-aALL
       val - 0=FDE and controller encryption (if HW supports) is allowed.
             1=Only support FDE encryption, disallow controller encryption.
MegaCli -AdpSetProp -PrCorrectUncfgdAreas -val -aN|-a0,1,2|-aALL
       val - 0= Correcting Media error during PR is disabled.
             1=Correcting Media error during PR is allowed.
MegaCli -AdpSetProp -DsblSpinDownHSP -val -aN|-a0,1,2|-aALL
       val - 0= Spinning down the Hot Spare is enabled.
             1=Spinning down the Hot Spare is disabled.
MegaCli -AdpSetProp -DefaultLdPSPolicy -Automatic| -None | -Maximum| -MaximumWithoutCaching -aN|-a0,1,2|-aALL
MegaCli -AdpSetProp -DisableLdPS -interval n1 -time n2 -aN|-a0,1,2|-aALL
       where n1 is the number of hours beginning at time n2
       where n2 is the number of minutes from 12:00am
MegaCli -AdpSetProp -ENABLEEGHSP -val -aN|-a0,1,2|-aALL
       val - 0= Disabled Emergency GHSP.
             1= Enabled Emergency GHSP.
MegaCli -AdpSetProp -ENABLEEUG -val -aN|-a0,1,2|-aALL
       val - 0= Disabled Emergency UG as Spare.
             1= Enabled Emergency UG as Spare.
MegaCli -AdpSetProp -ENABLEESMARTER -val -aN|-a0,1,2|-aALL
       val - 0= Disabled Emergency Spare as Smarter.
             1= Enabled Emergency Spare as Smarter.
MegaCli -AdpSetProp -DPMenable -val -aN|-a0,1,2|-aALL
       val - 0=Disable Drive Performance Monitoring .
             1=Enable Drive Performance Monitoring.
MegaCli -AdpSetProp -SupportSSDPatrolRead -val -aN|-a0,1,2|-aALL
       val - 0=Disable Patrol read for SSD drives .
             1=Enable Patrol read for SSD drives.
MegaCli -AdpGetProp CacheFlushInterval | RebuildRate | PatrolReadRate | ForceSGPIO
    | BgiRate | CCRate | ReconRate | SpinupDriveCount | SpinupDelay
    | CoercionMode | ClusterEnable | PredFailPollInterval | BatWarnDsbl
    | EccBucketSize | EccBucketLeakRate | EccBucketCount | AbortCCOnError
    | AlarmDsply | SMARTCpyBkEnbl | SSDSMARTCpyBkEnbl | NCQDsply
    | MaintainPdFailHistoryEnbl | RstrHotSpareOnInsert
    | EnblSpinDownUnConfigDrvs  | DisableOCR
    | BootWithPinnedCache | enblPI  |PreventPIImport | AutoEnhancedImportDsply | AutoDetectBackPlaneDsbl
    | CopyBackDsbl | LoadBalanceMode | UseFDEOnlyEncrypt | WBSupport | EnableJBOD
    | DsblCacheBypass | ExposeEnclDevicesEnbl | SpinDownTime | PrCorrectUncfgdAreas
    | UseDiskActivityforLocate | ENABLEEGHSP | ENABLEEUG | ENABLEESMARTER | Perfmode | PerfModeValues
    | -DPMenable -aN|-a0,1,2|-aALL
    | DefaultLdPSPolicy | DisableLdPsInterval | DisableLdPsTime | SpinUpEncDrvCnt
    | SpinUpEncDelay | PrCorrectUncfgdAreas
    | DsblSpinDownHSP | SupportSSDPatrolRead -aN|-a0,1,2|-aALL
MegaCli -AdpAllInfo -aN|-a0,1,2|-aALL  
MegaCli -AdpGetTime -aN|-a0,1,2|-aALL  
MegaCli -AdpSetTime yyyymmdd hh:mm:ss -aN   
MegaCli -AdpSetVerify -f fileName -aN|-a0,1,2|-aALL  
MegaCli -AdpBIOS -Enbl |-Dsbl | -SOE | -BE |  -HCOE | - HSM | EnblAutoSelectBootLd | DsblAutoSelectBootLd | -Dsply -aN|-a0,1,2|-aALL
MegaCli -AdpBootDrive {-Set {-Lx | -physdrv[E0:S0]}} | {-Unset {-Lx | -physdrv[E0:S0]}} |-Get -aN|-a0,1,2|-aALL
MegaCli -AdpAutoRbld -Enbl|-Dsbl|-Dsply -aN|-a0,1,2|-aALL
MegaCli -AdpCacheFlush -aN|-a0,1,2|-aALL
MegaCli -AdpPR -Dsbl|EnblAuto|EnblMan|Start|Suspend|Resume|Stop|Info|SSDPatrolReadEnbl |SSDPatrolReadDsbl  
         |{SetDelay Val}|{-SetStartTime yyyymmdd hh}|{maxConcurrentPD Val} -aN|-a0,1,2|-aALL
MegaCli -AdpCcSched -Dsbl|-Info|{-ModeConc | -ModeSeq [-ExcludeLD -LN|-L0,1,2]
   [-SetStartTime yyyymmdd hh ] [-SetDelay val ] } -aN|-a0,1,2|-aALL
MegaCli -AdpCcSched -SetStartTime yyyymmdd hh -aN|-a0,1,2|-aALL
MegaCli -AdpCcSched -SetDelay val  -aN|-a0,1,2|-aALL
MegaCli -FwTermLog -BBUoff|BBUoffTemp|BBUon|BBUGet|Dsply|Clear -aN|-a0,1,2|-aALL
MegaCli -AdpAlILog -aN|-a0,1,2|-aALL
MegaCli -AdpDiag [val] -aN|-a0,1,2|-aALL
          val - Time in second.
MegaCli -AdpGetPciInfo -aN|-a0,1,2|-aALL
MegaCli -AdpShutDown -aN|-a0,1,2|-aALL
MegaCli -AdpDowngrade -aN|-a0,1,2|-aALL
MegaCli -PDList -aN|-a0,1,2|-aALL
MegaCli -PDGetNum -aN|-a0,1,2|-aALL
MegaCli -pdInfo -PhysDrv[E0:S0,E1:S1,...] -aN|-a0,1,2|-aALL  
MegaCli -PDOnline  -PhysDrv[E0:S0,E1:S1,...] -aN|-a0,1,2|-aALL
MegaCli -PDOffline -PhysDrv[E0:S0,E1:S1,...] -aN|-a0,1,2|-aALL
MegaCli -PDMakeGood -PhysDrv[E0:S0,E1:S1,...] | [-Force] -aN|-a0,1,2|-aALL
MegaCli -PDMakeJBOD -PhysDrv[E0:S0,E1:S1,...] -aN|-a0,1,2|-aALL
MegaCli -PDHSP {-Set [-Dedicated [-ArrayN|-Array0,1,2...]] [-EnclAffinity] [-nonRevertible]}
     |-Rmv -PhysDrv[E0:S0,E1:S1,...] -aN|-a0,1,2|-aALL
MegaCli -PDRbld -Start|-Stop|-Suspend|-Resume|-ShowProg |-ProgDsply
        -PhysDrv [E0:S0,E1:S1,...] -aN|-a0,1,2|-aALL  
MegaCli -PDClear -Start|-Stop|-ShowProg |-ProgDsply
        -PhysDrv [E0:S0,E1:S1,...] -aN|-a0,1,2|-aALL  
MegaCli -PdLocate {[-start] | -stop} -physdrv[E0:S0,E1:S1,...] -aN|-a0,1,2|-aALL
MegaCli -PdMarkMissing -physdrv[E0:S0,E1:S1,...] -aN|-a0,1,2|-aALL
MegaCli -PdGetMissing -aN|-a0,1,2|-aALL
MegaCli -PdReplaceMissing -physdrv[E0:S0] -arrayA, -rowB -aN
MegaCli -PdPrpRmv [-UnDo] -physdrv[E0:S0] -aN|-a0,1,2|-aALL  
MegaCli -EncInfo -aN|-a0,1,2|-aALL
MegaCli -EncStatus -aN|-a0,1,2|-aALL
MegaCli -PhyInfo -phyM -aN|-a0,1,2|-aALL  
MegaCli -PhySetLinkSpeed -phyM -speed -aN|-a0,1,2|-aALL
MegaCli -PdFwDownload [offline][ForceActivate] {[-SataBridge] -PhysDrv[0:1] }|{-EncdevId[devId1]} -f -aN|-a0,1,2|-aALL
MegaCli -LDInfo -Lx|-L0,1,2|-Lall -aN|-a0,1,2|-aALL
MegaCli -LDSetProp  {-Name LdNamestring} | -RW|RO|Blocked|RemoveBlocked | WT|WB|ForcedWB [-Immediate] |RA|NORA|ADRA | DsblPI
        | Cached|Direct | -EnDskCache|DisDskCache | CachedBadBBU|NoCachedBadBBU
        -Lx|-L0,1,2|-Lall -aN|-a0,1,2|-aALL
MegaCli -LDSetPowerPolicy -Default| -Automatic| -None| -Maximum| -MaximumWithoutCaching
        -Lx|-L0,1,2|-Lall -aN|-a0,1,2|-aALL
MegaCli -LDGetProp  -Cache | -Access | -Name | -DskCache | -PSPolicy | Consistency -Lx|-L0,1,2|-LALL  
        -aN|-a0,1,2|-aALL
MegaCli -LDInit {-Start [-full]}|-Abort|-ShowProg|-ProgDsply -Lx|-L0,1,2|-LALL -aN|-a0,1,2|-aALL
MegaCli -LDCC {-Start [-force]}|-Abort|-Suspend|-Resume|-ShowProg|-ProgDsply -Lx|-L0,1,2|-LALL -aN|-a0,1,2|-aALL
MegaCli -LDBI -Enbl|-Dsbl|-getSetting|-Abort|-Suspend|-Resume|-ShowProg|-ProgDsply -Lx|-L0,1,2|-LALL -aN|-a0,1,2|-aALL  
MegaCli -LDRecon {-Start -rX [{-Add | -Rmv} -Physdrv[E0:S0,...]]}|-ShowProg|-ProgDsply
        -Lx -aN
MegaCli -LdPdInfo -aN|-a0,1,2|-aALL
MegaCli -LDGetNum -aN|-a0,1,2|-aALL
MegaCli -LDBBMClr -Lx|-L0,1,2,...|-Lall -aN|-a0,1,2|-aALL
MegaCli -getLdExpansionInfo -Lx|-L0,1,2|-Lall -aN|-a0,1,2|-aALL
MegaCli -LdExpansion -pN -dontExpandArray -Lx|-L0,1,2|-Lall -aN|-a0,1,2|-aALL
MegaCli -GetBbtEntries -Lx|-L0,1,2|-Lall -aN|-a0,1,2|-aALL
MegaCli -Cachecade -assign|-remove -Lx|-L0,1,2|-LALL -aN|-a0,1,2|-aALL
MegaCli -CfgLdAdd -rX[E0:S0,E1:S1,...] [WT|WB] [NORA|RA|ADRA] [Direct|Cached]
        [CachedBadBBU|NoCachedBadBBU] [-szXXX [-szYYY ...]]
        [-strpszM] [-Hsp[E0:S0,...]] [-AfterLdX] | [Secure]  
        [-Default| -Automatic| -None| -Maximum| -MaximumWithoutCaching] [-Cache] [-enblPI] [-Force]-aN
MegaCli -CfgCacheCadeAdd [-rX] -Physdrv[E0:S0,...] {-Name LdNamestring} [WT|WB|ForcedWB] [-assign -LX|L0,2,5..|LALL] -aN|-a0,1,2|-aALL
MegaCli -CfgEachDskRaid0 [WT|WB] [NORA|RA|ADRA] [Direct|Cached] [-enblPI]
        [CachedBadBBU|NoCachedBadBBU] [-strpszM]|[Secure] [-Default| -Automatic| -None| -Maximum| -MaximumWithoutCaching] [-Cache] -aN|-a0,1,2|-aALL
MegaCli -CfgClr [-Force] -aN|-a0,1,2|-aALL
MegaCli -CfgDsply -aN|-a0,1,2|-aALL
MegaCli -CfgCacheCadeDsply -aN|-a0,1,2|-aALL
MegaCli -CfgLdDel -LX|-L0,2,5...|-LALL [-Force] -aN|-a0,1,2|-aALL
MegaCli -CfgCacheCadeDel -LX|-L0,2,5...|-LALL -aN|-a0,1,2|-aALL
MegaCli -CfgFreeSpaceinfo -aN|-a0,1,2|-aALL
MegaCli -CfgSpanAdd -r10 -Array0[E0:S0,E1:S1] -Array1[E0:S0,E1:S1] [-ArrayX[E0:S0,E1:S1] ...]
        [WT|WB] [NORA|RA|ADRA] [Direct|Cached] [CachedBadBBU|NoCachedBadBBU]
        [-szXXX[-szYYY ...]][-strpszM][-AfterLdX]| [Secure]
        [-Default| -Automatic| -None| -Maximum| -MaximumWithoutCaching] [-Cache] [-enblPI] [-Force] -aN
MegaCli -CfgSpanAdd -r50 -Array0[E0:S0,E1:S1,E2:S2,...] -Array1[E0:S0,E1:S1,E2:S2,...]
        [-ArrayX[E0:S0,E1:S1,E2:S2,...] ...] [WT|WB] [NORA|RA|ADRA] [Direct|Cached]
        [CachedBadBBU|NoCachedBadBBU][-szXXX[-szYYY ...]][-strpszM][-AfterLdX]
        [Secure] [-Default| -Automatic| -None| -Maximum| -MaximumWithoutCaching] [-Cache] [-enblPI] [-Force] -aN
MegaCli -CfgSpanAdd -r60 -Array0[E0:S0,E1:S1,E2:S2,E3,S3...] -Array1[E0:S0,E1:S1,E2:S2,E3,S3...]
        [-ArrayX[E0:S0,E1:S1,E2:S2,E3,S3...] ...] [WT|WB] [NORA|RA|ADRA] [Direct|Cached]
        [CachedBadBBU|NoCachedBadBBU][-szXXX[-szYYY ...]][-strpszM][-AfterLdX]|
        [Secure] [-Default| -Automatic| -None| -Maximum| -MaximumWithoutCaching] [-Cache] [-enblPI] [-Force]-aN
MegaCli -CfgAllFreeDrv -rX [-SATAOnly] [-SpanCount XXX] [WT|WB] [NORA|RA|ADRA]
        [Direct|Cached] [CachedBadBBU|NoCachedBadBBU] [-strpszM]
        [-HspCount XX [-HspType -Dedicated|-EnclAffinity|-nonRevertible]]|
        [Secure] [-Default| -Automatic| -None| -Maximum| -MaximumWithoutCaching] [-Cache] [-enblPI] -aN
MegaCli -CfgSave -f filename -aN   
MegaCli -CfgRestore -f filename -aN   
MegaCli -CfgForeign -Scan | [-Passphrase sssssssssss] -aN|-a0,1,2|-aALL    
MegaCli -CfgForeign -Dsply [x] | [-Passphrase sssssssssss] -aN|-a0,1,2|-aALL    
MegaCli -CfgForeign -Preview [x] | [-Passphrase sssssssssss] -aN|-a0,1,2|-aALL    
MegaCli -CfgForeign -Import [x] | [-Passphrase sssssssssss] -aN|-a0,1,2|-aALL    
MegaCli -CfgForeign -Clear [x]|[-Passphrase sssssssssss] -aN|-a0,1,2|-aALL    
        x - index of foreign configurations. Optional. All by default.
MegaCli -AdpEventLog -GetEventLogInfo -aN|-a0,1,2|-aALL
MegaCli -AdpEventLog -GetEvents {-info -warning -critical -fatal} {-f } -aN|-a0,1,2|-aALL
MegaCli -AdpEventLog -GetSinceShutdown {-info -warning -critical -fatal} {-f } -aN|-a0,1,2|-aALL
MegaCli -AdpEventLog -GetSinceReboot {-info -warning -critical -fatal} {-f } -aN|-a0,1,2|-aALL
MegaCli -AdpEventLog -IncludeDeleted {-info -warning -critical -fatal} {-f } -aN|-a0,1,2|-aALL
MegaCli -AdpEventLog -GetLatest n {-info -warning -critical -fatal} {-f } -aN|-a0,1,2|-aALL
MegaCli -AdpEventLog -GetCCIncon -f -LX|-L0,2,5...|-LALL -aN|-a0,1,2|-aALL
MegaCli -AdpEventLog -Clear -aN|-a0,1,2|-aALL
MegaCli -AdpBbuCmd -aN|-a0,1,2|-aALL  
MegaCli -AdpBbuCmd -GetBbuStatus -aN|-a0,1,2|-aALL  
MegaCli -AdpBbuCmd -GetBbuCapacityInfo -aN|-a0,1,2|-aALL  
MegaCli -AdpBbuCmd -GetBbuDesignInfo -aN|-a0,1,2|-aALL  
MegaCli -AdpBbuCmd -GetBbuProperties -aN|-a0,1,2|-aALL  
MegaCli -AdpBbuCmd -BbuLearn -aN|-a0,1,2|-aALL  
MegaCli -AdpBbuCmd -BbuMfgSleep -aN|-a0,1,2|-aALL  
MegaCli -AdpBbuCmd -BbuMfgSeal -aN|-a0,1,2|-aALL  
MegaCli -AdpBbuCmd -getBbumodes  -aN|-a0,1,2|-aALL  
MegaCli -AdpBbuCmd -SetBbuProperties -f -aN|-a0,1,2|-aALL
MegaCli -AdpBbuCmd -GetGGEEPData offset [Hexaddress] NumBytes n -aN|-a0,1,2|-aALL
MegaCli -AdpBbuCmd -ScheduleLearn -Dsbl|-Info|[-STARTTIME DDD hh] -aN|-a0,1,2|-aALL
MegaCli -AdpFacDefSet -aN
MegaCli -AdpFwFlash -f filename [-ResetNow] [-NoSigChk] [-NoVerChk] [-FWTYPE n] -aN|-a0,1,2|-aALL  
MegaCli -AdpGetConnectorMode -ConnectorN|-Connector0,1|-ConnectorAll -aN|-a0,1,2|-aALL  
MegaCli -AdpSetConnectorMode -Internal|-External|-Auto -ConnectorN|-Connector0,1|-ConnectorAll -aN|-a0,1,2|-aALL  
MegaCli -PhyErrorCounters -aN|-a0,1,2|-aALL  
MegaCli -DirectPdMapping -Enbl|-Dsbl|-Dsply -aN|-a0,1,2|-aALL  
MegaCli -PDCpyBk -Start -PhysDrv[E0:S0,E1:S1] -aN|-a0,1,2|-aALL
MegaCli -PDCpyBk -Stop|-Suspend|-Resume|-ShowProg|-ProgDsply -PhysDrv[E0:S0] -aN|-a0,1,2|-aALL
MegaCli -PDInstantSecureErase -PhysDrv[E0:S0,E1:S1,...] | [-Force] -aN|-a0,1,2|-aALL
MegaCli -CfgSpanAdd -rX -array0[E0:S1,E1:S1.....] array1[E0:S1,E1:S1.....] -szxxx -enblPI -aN|-a0,1,2|-aALL
MegaCli -LDMakeSecure -Lx|-L0,1,2,...|-Lall -aN|-a0,1,2|-aALL
MegaCli -DeleteSecurityKey | [-Force] -aN
MegaCli -CreateSecurityKey -Passphrase sssssssssss [-KeyID kkkkkkkkkkk] -aN
MegaCli -CreateSecurityKey useEKMS -aN
MegaCli -ChangeSecurityKey -OldPassphrase sssssssssss | -Passphrase sssssssssss |
          [-KeyID kkkkkkkkkkk] -aN
MegaCli -ChangeSecurityKey -Passphrase sssssssssss |
          [-KeyID kkkkkkkkkkk] -aN
MegaCli -ChangeSecurityKey useEKMS -oldPassphrase sssssssssss -aN
MegaCli -ChangeSecurityKey -useEKMS -aN
MegaCli -GetKeyID [-PhysDrv[E0:S0]] -aN
MegaCli -SetKeyID -KeyID kkkkkkkkkkk -aN
MegaCli -VerifySecurityKey -Passphrase sssssssssss -aN
MegaCli -GetPreservedCacheList -aN|-a0,1,2|-aALL
MegaCli -DiscardPreservedCache -Lx|-L0,1,2|-Lall -force -aN|-a0,1,2|-aALL

       sssssssssss  - It must be between eight and thirty-two
                      characters and contain at least one number,
                      one lowercase letter, one uppercase
                      letter and one non-alphanumeric character.
       kkkkkkkkkkk -  Must be less than 256 characters.
MegaCli -ShowSummary [-f filename] -aN
MegaCli -ELF -GetSafeId -aN|-a0,1,2|-aALL
MegaCli -ELF -ControllerFeatures -aN|-a0,1,2|-aALL
MegaCli -ELF -Applykey key [Preview] -aN|-a0,1,2|-aALL
MegaCli -ELF -TransferToVault -aN|-a0,1,2|-aALL
MegaCli -ELF -DeactivateTrialKey -aN|-a0,1,2|-aALL
MegaCli -ELF -ReHostInfo -aN|-a0,1,2|-aALL
MegaCli -ELF -ReHostComplete -aN|-a0,1,2|-aALL
MegaCli -LDViewMirror -Lx|-L0,1,2,...|-Lall -aN|-a0,1,2|-aALL
MegaCli -LDJoinMirror -DataSrc [-force] -Lx|-L0,1,2,...|-Lall -aN|-a0,1,2|-aALL
MegaCli -SecureErase
    Start[
        Simple|
        [Normal   [ |ErasePattern ErasePatternA|ErasePattern ErasePatternA ErasePattern ErasePatternB]]|
        [Thorough [ |ErasePattern ErasePatternA|ErasePattern ErasePatternA ErasePattern ErasePatternB]]]
    | Stop
    | ShowProg
    | ProgDsply
    [-PhysDrv [E0:S0,E1:S1,...] | -Lx|-L0,1,2|-LALL] -aN|-a0,1,2|-aALL
MegaCli -Version -Cli|-Ctrl|-Driver|-Pd   -aN|-a0,1,2|-aALL
MegaCli -Perfmon {-start -interval } | {stop} | {-getresults -f } -aN
MegaCli -DpmStat -Dsply {lct | hist | ra | ext } [-physdrv[E0:S0]] -aN|-a0,1,2|-aALL  
MegaCli -DpmStat -Clear {lct | hist | ra | ext } -aN|-a0,1,2|-aALL  

    Note: The directly connected drives can be specified as [:S]

    Wildcard '?' can be used to specify the enclosure ID for the drive in the
      only enclosure without direct connected device or the direct connected
      drives with no enclosure in the system.

    Note:[-aALL] option assumes that the parameters specified are valid
       for all the Adapters.

    Note:ProgDsply option is not supported in VMWARE-COSLESS.

    The following options may be given at the end of any command above:

    [-Silent] [-AppLogFile filename] [-NoLog] [-page[N]]
                 [-] is optional.
                  N - Number of lines per page.
MegaCli XD -AddVd
MegaCli XD -RemVd
MegaCli XD -AddCdev | -force
MegaCli XD -RemCdev
MegaCli XD -VdList | -Configured | -Unconfigured
MegaCli XD -CdevList | -Configured | -Unconfigured
MegaCli XD -ConfigInfo
MegaCli XD -PerfStats
MegaCli XD -OnlineVd
MegaCli XD -WarpDriveInfo -iN | -iALL
MegaCli XD -FetchSafeId -iN | -iALL
MegaCli XD -ApplyActivationKey -iN

Exit Code: 0x00


MegaCLI Command Tutorial

This command lists all of the physical drives on adapter 0 or (-a0).

/opt/MegaRAID/MegaCli/MegaCli64  -pdlist -a0
 

Notable things

Slot Number:

Notice it will list the slot# so you know the physical position of the drive in the server.

Adapter #0

Enclosure Device ID: 32
Slot Number: 1
Enclosure position: 1
Device Id: 1
WWN: 5000C5003A260844
Sequence Number: 1
Media Error Count: 0
Other Error Count: 0
Predictive Failure Count: 0
Last Predictive Failure Event Seq Number: 0
PD Type: SAS

Raw Size: 279.396 GB [0x22ecb25c Sectors]
Non Coerced Size: 278.896 GB [0x22dcb25c Sectors]
Coerced Size: 278.875 GB [0x22dc0000 Sectors]
Sector Size:  0
Firmware state: Unconfigured(good), Spun Up
Device Firmware Level: FS64
Shield Counter: 0
Successful diagnostics completion on :  N/A
SAS Address(0): 0x5000c5003a260845
SAS Address(1): 0x0
Connected Port Number: 1(path0)
Inquiry Data: SEAGATE ST9300603SS     FS646SE3T176            
FDE Capable: Not Capable
FDE Enable: Disable
Secured: Unsecured
Locked: Unlocked
Needs EKM Attention: No
Foreign State: None
Device Speed: 6.0Gb/s
Link Speed: 6.0Gb/s
Media Type: Hard Disk Device
Drive Temperature :28C (82.40 F)
PI Eligibility:  No
Drive is formatted for PI information:  No
PI: No PI
Port-0 :
Port status: Active
Port's Linkspeed: 6.0Gb/s
Port-1 :
Port status: Active
Port's Linkspeed: Unknown
Drive has flagged a S.M.A.R.T alert : No

 

MegaRAID Create RAID 10 Example

First we need the slots of our drives, in this exampe on this server I only had 4 drives and knew I wanted each of them, so doing a grep on slot was fine to use any that came out.

Get Slot Numbers of Drives

MegaCli64 -PDList -aALL|grep -i slot
Slot Number: 1
Slot Number: 3
Slot Number: 5
Slot Number: 7

Use the slot numbers from above later on in our array creation.

Get "Enclosure Device ID":

MegaCli64 -PDList -aALL|grep -i enclosure
Enclosure Device ID: 32
Enclosure position: 1
Enclosure Device ID: 32
Enclosure position: 1
Enclosure Device ID: 32
Enclosure position: 1
Enclosure Device ID: 32
Enclosure position: 1

In our case we can see the enclosure ID is 32

Create RAID 10 Array

CfgSpanAdd command which is required for a RAID 10 array.

r10 specifies it as RAID 10

Array is required as RAID 10 in theory is 2 RAID 1's combined into a RAID 0.

Note that in the brackets the 32:1 is based on the fact from above that our enclosure ID is "32" and the 1 is one of our slot numbers of a disk we want to use.  Be sure to adjust according

MegaCli64 -CfgSpanAdd -r10 -Array0[32:1,32:3] -Array1[32:5,32:7] -a0
                                     
Adapter 0: Created VD 0

Adapter 0: Configured the Adapter!!

Exit Code: 0x00

Success in RAID 10

You will now see that you have a new device that automatically appears as the next available drive letter.

Disk /dev/sdc: 557.77 GiB, 598879502336 bytes, 1169686528 sectors
Disk model: PERC H710P      
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


MegaRAID RAID 0 Example

MegaCli64 -CfgLdAdd -r0[32:5] -a0

The above creates a RAID 0 array out of a single drive located in enclosure 32 and slot#5 on adapter 0.

This is the way if you want simpler and better performance by using mdadm to create your own array.

Of course you could have specified multiple drives if you truly wanted a fast but totally unreliable RAID 0 multi disk array (eg. -r0[32:5,32:4]

Check array/rebuild state:

./MegaCli64  -LDInfo -LALL -aALL
                                     

Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 557.75 GB
Sector Size         : 512
Mirror Data         : 557.75 GB
State               : Optimal
Strip Size          : 64 KB
Number Of Drives per span:2
Span Depth          : 2
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Ongoing Progresses:
  Background Initialization: Completed 24%, Taken 10 min.
Encryption Type     : None
Default Power Savings Policy: Controller Defined
Current Power Savings Policy: None
Can spin up in 1 minute: Yes
LD has drives that support T10 power conditions: Yes
LD's IO profile supports MAX power savings with cached writes: No
Bad Blocks Exist: No
Is VD Cached: Yes
Cache Cade Type : Read Only


MegaRAID Create Single Disk RAID 0 Example

If you don't want to use the entire RAID function and want to use mdadm, you can create a fake RAID 0 of each drive to expose it as a sort of normal drive and then make a RAID out of that for use with mdadm.  mdadm usually gives better performance, especially in RAID 10.

For example using an old 4 disk RAID 10 array with MegaRAID it produces only 249MB/s:

5242880000 bytes (5.2 GB, 4.9 GiB) copied, 21.0175 s, 249 MB/s

Now compare the same 4 disks with mdadm RAID 10 with virtual raid 0 drives

5242880000 bytes (5.2 GB, 4.9 GiB) copied, 16.2648 s, 322 MB/s
Note that this test was done while the array was still initializing too!

Look at how much faster the fully sync'd mdadm RAID 10 is (nearly 2x faster than the MegaRAID 10):

5242880000 bytes (5.2 GB, 4.9 GiB) copied, 10.412 s, 504 MB/s
 

Create each PD/physical drive using the enlosure ID and slot ID [32:1] for example.

MegaCli64 CfgLdAdd -r0 [32:1] -a0
                                     
Adapter 0: Created VD 0

Adapter 0: Configured the Adapter!!

Exit Code: 0x00

MegaCli64 CfgLdAdd -r0 [32:3] -a0
                                     
Adapter 0: Created VD 1

Adapter 0: Configured the Adapter!!

Exit Code: 0x00

MegaCli64 CfgLdAdd -r0 [32:5] -a0
                                     
Adapter 0: Created VD 2

Adapter 0: Configured the Adapter!!

Exit Code: 0x00

MegaCli64 CfgLdAdd -r0 [32:7] -a0
                                     
Adapter 0: Created VD 3

Adapter 0: Configured the Adapter!!

After this just do the normal mdadm config on each disk (in my case the above created a /dev/sdc sdd sde sdf).

How To Delete Virtual Drive

For example above we created the Virtual Drive with ID "0"

Virtual Drive: 0 (Target Id: 0)

./MegaCli64 CfgLdDel -L0 -Force -a0

The 0 in -L0 means logical drive 0 from above.  Change to match your ID.

If you don't use -Force before -a0 you get this error:  Virtual Disk is associate with Cache Cade. Please Use force option to delete

                                     
Adapter 0: Deleted Virtual Drive-0(target id-0)
 

How to use smartctl normally with megaraid

One other irritating thing about tools like megaraid is that smart doesn't work as you expect.  Take a virtual drive /dev/sdc it doesn't show anything useful.

With smartctl you can use the megaraid option/plugin and specify the drive#/slot# and get normal info like this:

Note below the magic is in -d megaraid,3 which says we want the megaraid drive#3 or slot#3 smart info, which then gives us the normal expected info.

 smartctl -a /dev/sdc -d megaraid,3 -T permissive
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-26-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               IBM-ESXS
Product:              ST9300603SS   F
Revision:             B53A
Compliance:           SPC-3
User Capacity:        300,000,000,000 bytes [300 GB]
Logical block size:   512 bytes
Rotation Rate:        10000 rpm
Form Factor:          2.5 inches
Logical Unit id:      0x5000c5001dbdea07
Serial number:        3SE1MFK400009035N4B3
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Mon Mar 24 20:16:16 2025 UTC
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature:     0 C
Drive Trip Temperature:        0 C

Elements in grown defect list: 0

Error Counter logging not supported

Device does not support Self Test logging
 

Were you trying to convert a PDF and get this message?:

Convert-im6.q16: attempt to perform an operation not allowed by the security policy `PDF' @ error/constitute.c/IsCoderAuthorized/413.
convert-im6.q16: no images defined `pts-time.jpg' @ error/convert.c/ConvertImageCommand/3258.
 

Solution:

Find the "PDF" pattern and set it like below:

  <policy domain="coder" rights="read|write" pattern="PDF" />
 

By default the rights are none (rights="none") which is why you get that error.

We've had clients asking why their CDN is not working, it is often a PHP setting that causes the below header to be sent:

  expires: Thu, 19 Nov 1981 08:52:00 GMT
 

Solution Edit your /etc/php.ini

Set the option below as just being empty.  Generally the default is nocache and will result in sending the expires header from 1981.

session.cache_limiter =

Here is what the man says about the option:

; Set to {nocache,private,public,} to determine HTTP caching aspects
; or leave this empty to avoid sending anti-caching headers.
; http://php.net/session.cache-limiter
 

However, we have often seen that any option but being empty results in the expires header being sent.  If you want your content to be cachable by CDN, then make sure the session.cache_limiter contains a NULL/empty value.

Step 1.) Install virt-manager

sudo apt install virt-manager

Step 2.) Start libvirtd

sudo systemctl start libvirtd

sudo systemctl enable libvirtd

Step 3.) Permissions

Your user needs access to libvirt and kvm or it won't work without running as sudo.

sudo usermod -a -G kvm yourusername


sudo usermond -a -G libvirt yourusername

Step 4.) Logout and Login

If you get errors relating to being unable to connect to QEMU or not active connection, it is probably a permissions issue. 

if you get errors with being unable to connect to libvirtd, it is probably not started.

A practical guide for admins who need to plan the required amount of bandwidth and what connectors/cards are needed for which.

These errors are usually caused by the lack mysql client dev files

If using mariadb install this:

apt-get install libmariadbclient-dev

If using mysql install this:

apt-get install libmysqlclient-dev

pip3 install mysql
The directory '/root/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/root/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting mysql
  Downloading https://files.pythonhosted.org/packages/9a/52/8d29c58f6ae448a72fbc612955bd31accb930ca479a7ba7197f4ae4edec2/mysql-0.0.3-py3-none-any.whl
Collecting mysqlclient (from mysql)
  Downloading https://files.pythonhosted.org/packages/50/5f/eac919b88b9df39bbe4a855f136d58f80d191cfea34a3dcf96bf5d8ace0a/mysqlclient-2.1.1.tar.gz (88kB)
    100% |████████████████████████████████| 92kB 5.5MB/s
    Complete output from command python setup.py egg_info:
    /bin/sh: 1: mysql_config: not found
    /bin/sh: 1: mariadb_config: not found
    /bin/sh: 1: mysql_config: not found
    mysql_config --version
    mariadb_config --version
    mysql_config --libs
    Traceback (most recent call last):
      File "", line 1, in
      File "/tmp/pip-install-twfzngc5/mysqlclient/setup.py", line 15, in
        metadata, options = get_config()
      File "/tmp/pip-install-twfzngc5/mysqlclient/setup_posix.py", line 70, in get_config
        libs = mysql_config("libs")
      File "/tmp/pip-install-twfzngc5/mysqlclient/setup_posix.py", line 31, in mysql_config
        raise OSError("{} not found".format(_mysql_config_path))
    OSError: mysql_config not found
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-twfzngc5/mysqlclient/


Make sure you choose your Trunk under "Trunk Sequence for Matched Routes"

 Setup Your Dial-Patterns Using the Wizard

 Remember to Click Submit and then Apply

Generally most containers are by default set not to start automatically.

Ther eare 3 settings for the "RestartPolicy" of containers:

• No: Do not automatically restart the container (default).
• Always: Always restart the container regardless of the exit status.
• Unless-stopped: Always restart the container unless it is explicitly stopped.
• On-failure: Restart the container only if it exits with a non-zero status

Here is how you can see the policy:

docker ps -aq | xargs -I {} docker inspect --format='{{.Name}}: {{.HostConfig.RestartPolicy.Name}}' {}
 

We can see the output of all containers on the node and in the case below we see that all 3 are set to "no".

/festive_wiles: no
/magical_goldberg: no
/angry_hamilton: no

Say if we wanted "festive_wiles" to always be running (eg. if the server/node is rebooted we want the container to startup automatically):

docker update --restart=always festive_wiles

How can set these startup/restart settings when creating the container?

We just specify --restart=

docker run --name YourContainerName --restart=Always -d nginx

Step 1 - Login to FreePBX and Go to Connectivity

Click Connectivity -> Extensions

Step 2 - Create Extension

Click on "Quick Create Extension" or "Add Extension".

Quick is fine for more users, unless you have more specific requirements.

Step 3 - Create the Extension

Once you click finish, the details are then e-mailed to the user.

If you use Quick Create, you can just edit the extension and click on their "secret" field to find the password.

To make your SIP softphone or real register use the following info:

SIP Server: Your IP/Domain of FreePBX

Username: Extension# (eg 1234)

Password: the secret

3CX Example

Disk /dev/sdb: 15.22 GiB, 16336814080 bytes, 31907840 sectors
Disk model: SD/MMC          
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

 

The fdisk output is above and below is the error you may get when trying to use the drive or even format it.

Warning: The driver descriptor says the physical block size is 2048 bytes, but Linux says it is 512 bytes.

wipefs is a useful tool to show us what filesystems exist on the drive.

root@mint:/home/mint# wipefs /dev/sdb
DEVICE OFFSET TYPE UUID      LABEL
sdb    0x52   vfat 53E2-5BC6
sdb    0x0    vfat 53E2-5BC6
sdb    0x1fe  vfat 53E2-5BC6
 

Let's wipe out all filesystems to fix the error

Make sure you have backed up any data. 

Change sdX below with the actual drive you want to do this to.

wipefs --all /dev/sdX

Most relevant config points from my video here.

1.) Create a new End User

The most important part is setting the "Digest Credentials", that is the password that the phone will use to authenticate.

2.) Add a New Phone

Device -> Phone

3.) 

Set a fake MAC address in the right format eg. AAABBBCCCDDD.

Remember to assign the phone to the user you created earlier.

Set the "Third-party SIP Device Basic - Standard SIP" security profile as below.

Scroll down and set the Digest user to the user we sent earlier.

Remember to create a DN, your phone will then authenticate with the DN as the username and the password of the "Digest User".

Before starting you should reset your phone(s) if they have been previously registered to another CME/CUCM:

https://realtechtalk.com/Cisco_IP_Phone_How_To_Reset_To_Factory_Settings_Instructions_CP_9971_8800_8900_8845_8851_8841_8831_7961_7960_7945_7942_7941_7940-2212-articles

https://realtechtalk.com/How_to_Reset_CIPC_Cisco_IP_Communicator_for_CME_CUCM_CallManager-2503-articles

Before starting go to "Cisco Unified Serviceability" drop-down on the top-right

Under Tools go to Service Activation.

Enable "CallManger" and "tftp" in order for phones to register.

Step 1 - Go The Phone Menu

Step 2 - Add New Phone

Step 3 - Choose Phone Type

Step 4 - Set Phone MAC

The format is SEPMACADDRESS

As we can see, the minimum requirements are that we specify the following:

Device Name:
Device Pool:
Owner User ID:
Device Security Profile:

Step 5 - Add New DN (top left)

Although it is well-known that pptp is not secure and is subject to many forms of attacks, the reality is that a lot of legacy and embedded devices use pptp.  I argue that if it is being used for routing or remote access or over an already secure connection (eg. another VPN like ikev2) then this is still acceptable.  Or in a LAN or in a public environment where no private data is exchanged.  However, if the nature of the data is extremely sensitive, you should do whatever it takes to have the second layer of encryption by using a secure VPN protocol.

In iptables you can find many threads and discussions about how to make pptp work with iptables, with crazy forwarding rules and blindly and manually allowing all GRE etc...  However it's much more simple in my experience.  You just need to enable the netfilter conntracking to be able to connect your pptp client.

This solution also applies to a node running Kubernetes and Docker containers (eg. an embedded device that is for some odd reason using pptp).  If you can, switch to ikev2 or OpenVPN.

sysctl -w net.netfilter.nf_conntrack_helper=1

For permanent changes add this to /etc/sysctl.conf:

net.netfilter.nf_conntrack_helper=1

There are other modules necessary to make pptp work, but you can see that any recent kernel will  load them on its own:

nf_nat_pptp            20480  0
nf_conntrack_pptp      24576  1 nf_nat_pptp
nf_nat                 45056  3 nf_nat_pptp,iptable_nat,xt_MASQUERADE
nf_conntrack          139264  6 xt_conntrack,nf_nat,nf_conntrack_pptp,nf_nat_pptp,nf_conntrack_netlink,xt_MASQUERADE


Sometimes systemd-journald can take several hundred megs of RAM or more which is bad for microservices and embedded devices.

Edit /etc/systemd/journald.conf

You can set it to max 5M of RAM like below:

SystemMaxUse=5M
 

FreePBX official install guide is here.

Requirements:

1. Debian 12 Download Link
2. Minimal - System Utilities
3. RAM: 4G
4. HDD: 20G

Note that if you don't have the required base OS you will get an error like this at the end of the install

2024-10-23 00:30:22 - Upgrading FreePBX 17 modules
2024-10-23 00:30:22 - Installation failed at step Upgrading FreePBX 17 modules. Please check log /var/log/pbx/freepbx17-install-2024.11.05-00.14.36.log for details.
2024-10-23 00:30:22 - Error at line: 1135 exiting with code 255 (last command was: fwconsole ma upgradeall >> $log)
2024-10-23 00:30:22 - Exiting script


#log

In modulefunctions.class.php line 2814:
                                                                                              
  preg_replace(): Passing null to parameter #3 ($subject) of type array|string is deprecated  
                                                                                              

moduleadmin [-f|--force] [-d|--debug] [--edge] [--ignorecache] [--stable] [--color] [--skipchown] [-e|--autoenable] [--skipdisabled] [--snapshot SNAPSHOT] [--format FORMAT] [-R|--repo REPO] [-t|--tag TAG] [--skipbreakingcheck] [--sendemail] [--onlystdout] [--] [...]

2024-10-23 00:30:22 - ****** INSTALLATION FAILED *****
2024-10-23 00:30:22 - Installation failed at step Upgrading FreePBX 17 modules. Please check log /var/log/pbx/freepbx17-install-2024.11.05-00.14.36.log for details.
2024-10-23 00:30:22 - Error at line: 1135 exiting with code 255 (last command was: fwconsole ma upgradeall >> $log)
2024-10-23 00:30:22 - Exiting script

Step 1- Get and execute the official Sangoma (maker of Asterisk) install script:

wget https://github.com/FreePBX/sng_freepbx_debian_install/raw/master/sng_freepbx_debian_install.sh

bash sng_freepbx_debian_install.sh

The install took about 20 minutes for me on a single core Xeon on an SSD.

If successful it should look like this at the end:

______                   ______ ______ __   __
|  ___|                  | ___ | ___ \ / /
| |_    _ __   ___   ___ | |_/ /| |_/ / V /
|  _|  | '__| / _ / _ |  __/ | ___ /  
| |    | |   |  __/|  __/| |    | |_/ // /^
_|    |_|    ___| ___|_|    ____/ /   /
                                              
                                              
NOTICE! You have 4 notifications! Please log into the UI to see them!
Current Network Configuration
+-----------+-------------------+---------------------------+
| Interface | MAC Address       | IP Addresses              |
+-----------+-------------------+---------------------------+
| ens3      | |               |
|           |                   | fe80::dcad:beff:feef:6e79 |
+-----------+-------------------+---------------------------+

Please note most tasks should be handled through the GUI.
You can access the GUI by typing one of the above IPs in to your web browser.
For support please visit:
    http://www.freepbx.org/support-and-professional-services

+---------------------------------------------------------------------+
| This machine is not activated.  Activating your system ensures that |
| your machine is eligible for support and that it has the ability to |
| install Commercial Modules.                                         |
|                                                                     |
| If you already have a Deployment ID for this machine, simply run:   |
|                                                                     |
|    fwconsole sysadmin activate deploymentid                         |
|                                                                     |
| to assign that Deployment ID to this system. If this system is new, |
| please go to Activation (which is on the System Admin page in the   |
| Web UI) and create a new Deployment there.                          |
+---------------------------------------------------------------------+
 

Step 2 - Setup and Login to the GUI

 Then enter the Administration Section

Do all of the default Firewall Settings and then you'll be in the Admin home page

Firewall Issues

CLI Firewall Commands:

fwconsole firewall help
Valid Commands:
disable : Disable the System Firewall. This will shut it down cleanly.
stop : Stop the System Firewall
start : Start (and enable, if disabled) the System Firewall
restart : Restart the System Firewall
lerules [enable] or [disable] : Enable or disable Lets Encrypt rules.
trust : Add the hostname or IP specified to the Trusted Zone
untrust : Remove the hostname or IP specified from the Trusted Zone
list [zone] : List all entries in zone 'zone'
add [zone] [id id id..] : Add to 'zone' the IDs provided.
del [zone] [id id id..] : Delete from 'zone' the IDs provided.
listzones : Show zones that can be used to add and del.
fix_custom_rules : Create the files for the custom rules if they don't exist and set the permissions and owners correctly.
sync : Synchronizes all selected zones of the firewall module with the intrusion detection whitelist.
f2bstatus or f2bs : Display ignored and banned IPs. (Only root user).

When adding or deleting from a zone, one or many IDs may be provided.
These may be IP addresses, hostnames, or networks.
 

Example:

fwconsole firewall add trusted 10.46.80.0/24 hostname.example.com 1.2.3.4

Note that the firewall in production should never be disabled as there are massive amounts of hackers that target FreePBX and SIP servers.  This can be used for learning, but ideally the firewall should be configured to whitelist yourself or other trusted IPs, rather than completely disabling.

If you are locked out:

Change 192.168.1.0/24 to your subnet or IP

iptables -I INPUT -s 192.168.1.0/24 -j ACCEPT


systemctl stop fail2ban

This buys time before the firewall reactivates.

Bug in installer script:

This happens if you try the near impossible by using an older/different Debian

The current script as of 2024-10-11 has a bug where they pass -S to add-apt-repository when it must be a lower case -s which breaks everything as you can see in the log:

 esolving deb.freepbx.org (deb.freepbx.org)... 52.217.40.28, 54.231.128.57, 52.217.9.44, ...
Connecting to deb.freepbx.org (deb.freepbx.org)|52.217.40.28|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3139 (3.1K) [binary/octet-stream]
Saving to: 'STDOUT'

     0K ...                                                   100% 19.4M=0s

2024-10-11 21:52:03 (19.4 MB/s) - written to stdout [3139/3139]

Usage: add-apt-repository

add-apt-repository is a script for adding apt sources.list entries.
It can be used to add any repository and also provides a shorthand
syntax for adding a Launchpad PPA (Personal Package Archive)
repository.

- The apt repository source line to add. This is one of:
  a complete apt line in quotes,
  a repo url and areas in quotes (areas defaults to 'main')
  a PPA shortcut.
  a distro component

  Examples:
    apt-add-repository 'deb http://myserver/path/to/repo stable myrepo'
    apt-add-repository 'http://myserver/path/to/repo myrepo'
    apt-add-repository 'https://packages.medibuntu.org free non-free'
    apt-add-repository http://extras.ubuntu.com/ubuntu
    apt-add-repository ppa:user/repository
    apt-add-repository ppa:user/distro/repository
    apt-add-repository multiverse

If --remove is given the tool will remove the given sourceline from your
sources.list


add-apt-repository: error: no such option: -S
 

This can be fixed by editing the bash installer file you downloaded and changing -S to -s

 This is a quick overview of the main screens that you will need to go through to do the Cisco Unified Communication Manager installation (CUCM). 

Skip the media test

Skip the Platform Installation Wizard (we'll do it later)

Set the relevant hostname and IP address information

Remember below is just an example, make sure you set a valid IP for your network.

Don't worry about a DNS error exactly like below

If you get the exact error below notice that it is complaining about "Reverse DNS lookup failed".  You can safely proceed and continue the install, or if you control your own DNS, then you can add a reverse entry for your IP to avoid this error.

In the next screens you will be creating different usernames and passwords for different purposes (eg. Admin user, App/UI user etc..).  Make sure you write down what each username and password is for and DO NOT forget them .

Create your SSL certificate, normally it would be something relevant to your company eg. (yourcompany.com).

First Node Configuration Screen.

If this is your first install and you don't have an existing Cluster you should choose "Yes".

Otherwise you would Choose "No" and enter the details of the Publisher in the next screen.

Normally we should have NTP in production, you may opt not to use NTP if your firewall blocks it or if your network has no outside access.

Keep your Security Password Somewhere Safe

The most common issue in setting up a Cluster is that you have bad network and/or the wrong Security Password.

Normally you would want SMTP but if you are on a network that doesn't allow SMTP or you have no mail account to use, it is OK to say "No" for now.

This step is another crucial part of the install and where people go wrong.

Note that if you have specified a bad IP or a non-working default gateway that this step will fail and you'll need to go back and set proper IP info.

This is one of the longest steps below.  Note that there is some timeout in the install script, if this step takes too long the install times out, fails and you have to reinstall. 

This is normally only an issue on machines with lots of IOPS of other usage or if you have an older/slower RAID array or mechanical disks/insufficient IO bandwidth at the moment.

Congrats!  After this point, you should be able to reboot and login to the Web GUI of CUCM (using your application user).

Note that it may take 5-15 minutes on initial boot for the Web GUI to be ready.

Just use pdfimages:

Change yourpdf.pdf to the name of your pdf

output-name is the name format (eg. output-name01.jpg output-name02.jpg) is how the files will be named if you use "output-name".  Change as needed.

pdfimages -all yourpdf.pdf output-name

After that you will have extracted all of the images.

You can also just run the command with -list and see all of the images contained within the pdf like this:

pdfimages -list file.pdf

page   num  type   width height color comp bpc  enc interp  object ID x-ppi y-ppi size ratio
--------------------------------------------------------------------------------------------
   1     0 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
   2     1 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
   2     2 image    1120  1120  rgb     3   8  jpeg   no         8  0   136   136  245K 6.7%
   2     3 smask    1120  1120  gray    1   8  image  no         8  0   136   136 1236B 0.1%
   3     4 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
   3     5 image    1120  1120  rgb     3   8  jpeg   no        14  0   136   136  216K 5.9%
   3     6 smask    1120  1120  gray    1   8  image  no        14  0   136   136 1236B 0.1%
   4     7 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
   5     8 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
   6     9 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
   7    10 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
   8    11 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
   9    12 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  10    13 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  11    14 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  11    15 image     726   429  rgb     3   8  image  no        41  0    66    66 13.6K 1.5%
  11    16 smask     726   429  gray    1   8  image  no        41  0    66    66  343B 0.1%
  12    17 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  12    18 image     726   429  rgb     3   8  image  no        48  0    66    66 12.4K 1.4%
  12    19 smask     726   429  gray    1   8  image  no        48  0    66    66  343B 0.1%
  13    20 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  13    21 image     726   429  rgb     3   8  image  no        55  0    66    66 10.6K 1.2%
  13    22 smask     726   429  gray    1   8  image  no        55  0    66    66  343B 0.1%
  14    23 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  14    24 image     726   429  rgb     3   8  image  no        62  0    66    66 13.1K 1.4%
  14    25 smask     726   429  gray    1   8  image  no        62  0    66    66  343B 0.1%
  15    26 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  15    27 image     726   429  rgb     3   8  image  no        69  0    66    66 11.9K 1.3%
  15    28 smask     726   429  gray    1   8  image  no        69  0    66    66  343B 0.1%
  16    29 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  16    30 image     726   429  rgb     3   8  image  no        76  0    66    66 16.2K 1.8%
  16    31 smask     726   429  gray    1   8  image  no        76  0    66    66  343B 0.1%
  17    32 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  17    33 image     726   429  rgb     3   8  image  no        83  0    66    66 10.8K 1.2%
  17    34 smask     726   429  gray    1   8  image  no        83  0    66    66  343B 0.1%
  18    35 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  18    36 image     726   429  rgb     3   8  image  no        90  0    66    66 11.2K 1.2%
  18    37 smask     726   429  gray    1   8  image  no        90  0    66    66  343B 0.1%
  19    38 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  19    39 image     726   429  rgb     3   8  image  no        97  0    66    66 13.3K 1.5%
  19    40 smask     726   429  gray    1   8  image  no        97  0    66    66  343B 0.1%
  20    41 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  20    42 image     726   429  rgb     3   8  image  no       104  0    66    66 13.4K 1.5%
  20    43 smask     726   429  gray    1   8  image  no       104  0    66    66  343B 0.1%
  21    44 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  21    45 image     726   429  rgb     3   8  image  no       111  0    66    66 8856B 0.9%
  21    46 smask     726   429  gray    1   8  image  no       111  0    66    66  343B 0.1%
  22    47 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  22    48 image     726   429  rgb     3   8  image  no       118  0    66    66 8841B 0.9%
  22    49 smask     726   429  gray    1   8  image  no       118  0    66    66  343B 0.1%
  23    50 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  23    51 image     726   429  rgb     3   8  image  no       125  0    66    66 17.3K 1.9%
  23    52 smask     726   429  gray    1   8  image  no       125  0    66    66  343B 0.1%
  24    53 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  24    54 image     726   429  rgb     3   8  image  no       132  0    66    66 15.0K 1.6%
  24    55 smask     726   429  gray    1   8  image  no       132  0    66    66  343B 0.1%
  25    56 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  25    57 image     726   429  rgb     3   8  image  no       139  0    66    66 15.0K 1.6%
  25    58 smask     726   429  gray    1   8  image  no       139  0    66    66  343B 0.1%
  26    59 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  26    60 image     726   429  rgb     3   8  image  no       146  0    66    66 14.8K 1.6%
  26    61 smask     726   429  gray    1   8  image  no       146  0    66    66  343B 0.1%
  27    62 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  27    63 image     726   429  rgb     3   8  image  no       153  0    66    66 13.6K 1.5%
  27    64 smask     726   429  gray    1   8  image  no       153  0    66    66  343B 0.1%
  28    65 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  28    66 image     726   429  rgb     3   8  image  no       160  0    66    66 15.0K 1.6%
  28    67 smask     726   429  gray    1   8  image  no       160  0    66    66  343B 0.1%
  29    68 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  29    69 image     726   429  rgb     3   8  image  no       167  0    66    66 13.3K 1.5%
  29    70 smask     726   429  gray    1   8  image  no       167  0    66    66  343B 0.1%
  30    71 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  30    72 image     726   429  rgb     3   8  image  no       174  0    66    66 12.9K 1.4%
  30    73 smask     726   429  gray    1   8  image  no       174  0    66    66  343B 0.1%
  31    74 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  31    75 image     726   429  rgb     3   8  image  no       181  0    66    66 13.3K 1.5%
  31    76 smask     726   429  gray    1   8  image  no       181  0    66    66  343B 0.1%
  32    77 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  32    78 image     726   429  rgb     3   8  image  no       188  0    66    66 13.7K 1.5%
  32    79 smask     726   429  gray    1   8  image  no       188  0    66    66  343B 0.1%
  33    80 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  33    81 image     726   429  rgb     3   8  image  no       195  0    66    66 12.3K 1.3%
  33    82 smask     726   429  gray    1   8  image  no       195  0    66    66  343B 0.1%
  34    83 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  34    84 image     726   429  rgb     3   8  image  no       202  0    66    66 12.4K 1.4%
  34    85 smask     726   429  gray    1   8  image  no       202  0    66    66  343B 0.1%
  35    86 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  35    87 image     726   429  rgb     3   8  image  no       209  0    66    66 13.2K 1.5%
  35    88 smask     726   429  gray    1   8  image  no       209  0    66    66  343B 0.1%
  36    89 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  36    90 image     726   429  rgb     3   8  image  no       216  0    66    66 12.9K 1.4%
  36    91 smask     726   429  gray    1   8  image  no       216  0    66    66  343B 0.1%
  37    92 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  37    93 image     726   429  rgb     3   8  image  no       223  0    66    66 13.2K 1.5%
  37    94 smask     726   429  gray    1   8  image  no       223  0    66    66  343B 0.1%
  38    95 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  38    96 image     726   429  rgb     3   8  image  no       230  0    66    66 12.3K 1.4%
  38    97 smask     726   429  gray    1   8  image  no       230  0    66    66  343B 0.1%
  39    98 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  39    99 image     726   429  rgb     3   8  image  no       237  0    66    66 12.5K 1.4%
  39   100 smask     726   429  gray    1   8  image  no       237  0    66    66  343B 0.1%
  40   101 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  40   102 image     726   429  rgb     3   8  image  no       244  0    66    66 12.4K 1.4%
  40   103 smask     726   429  gray    1   8  image  no       244  0    66    66  343B 0.1%
  41   104 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  41   105 image     726   429  rgb     3   8  image  no       251  0    66    66 12.5K 1.4%
  41   106 smask     726   429  gray    1   8  image  no       251  0    66    66  343B 0.1%
  42   107 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  42   108 image     726   429  rgb     3   8  image  no       258  0    66    66 17.3K 1.9%
  42   109 smask     726   429  gray    1   8  image  no       258  0    66    66  343B 0.1%
  43   110 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  43   111 image     726   429  rgb     3   8  image  no       265  0    66    66 14.7K 1.6%
  43   112 smask     726   429  gray    1   8  image  no       265  0    66    66  343B 0.1%
  44   113 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  44   114 image     726   429  rgb     3   8  image  no       272  0    66    66 15.5K 1.7%
  44   115 smask     726   429  gray    1   8  image  no       272  0    66    66  343B 0.1%
  45   116 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  45   117 image     726   429  rgb     3   8  image  no       279  0    66    66 12.8K 1.4%
  45   118 smask     726   429  gray    1   8  image  no       279  0    66    66  343B 0.1%
  46   119 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  46   120 image     726   429  rgb     3   8  image  no       286  0    66    66 15.0K 1.6%
  46   121 smask     726   429  gray    1   8  image  no       286  0    66    66  343B 0.1%
  47   122 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  47   123 image     726   429  rgb     3   8  image  no       293  0    66    66 14.4K 1.6%
  47   124 smask     726   429  gray    1   8  image  no       293  0    66    66  343B 0.1%
  48   125 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  48   126 image     726   429  rgb     3   8  image  no       300  0    66    66 15.6K 1.7%
  48   127 smask     726   429  gray    1   8  image  no       300  0    66    66  343B 0.1%
  49   128 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  49   129 image     726   429  rgb     3   8  image  no       307  0    66    66 13.6K 1.5%
  49   130 smask     726   429  gray    1   8  image  no       307  0    66    66  343B 0.1%
  50   131 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  50   132 image     726   429  rgb     3   8  image  no       314  0    66    66 14.3K 1.6%
  50   133 smask     726   429  gray    1   8  image  no       314  0    66    66  343B 0.1%
  51   134 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  51   135 image     726   429  rgb     3   8  image  no       321  0    66    66 15.2K 1.7%
  51   136 smask     726   429  gray    1   8  image  no       321  0    66    66  343B 0.1%
  52   137 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  52   138 image     726   429  rgb     3   8  image  no       328  0    66    66 14.7K 1.6%
  52   139 smask     726   429  gray    1   8  image  no       328  0    66    66  343B 0.1%
  53   140 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  53   141 image     726   429  rgb     3   8  image  no       335  0    66    66 12.3K 1.3%
  53   142 smask     726   429  gray    1   8  image  no       335  0    66    66  343B 0.1%
  54   143 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  54   144 image     726   429  rgb     3   8  image  no       342  0    66    66 12.5K 1.4%
  54   145 smask     726   429  gray    1   8  image  no       342  0    66    66  343B 0.1%
  55   146 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  55   147 image     726   429  rgb     3   8  image  no       349  0    66    66 12.3K 1.3%
  55   148 smask     726   429  gray    1   8  image  no       349  0    66    66  343B 0.1%
  56   149 image    2543  1553  rgb     3   8  jpeg   no         4  0   231   188 64.6K 0.6%
  56   150 image     726   429  rgb     3   8  image  no       356  0    66    66 12.8K 1.4%
  56   151 smask     726   429  gray    1   8  image  no       356  0    66    66  343B 0.1%


In Windows, with some NICs, especially Intel, Windows may enable features on the card that break it in Linux.  It is hard to troubleshoot as what you'll see is that the NIC is still detected in Linux, the NIC/port will be up but nothing will work (eg. DHCP requests or even static IPs won't work).  You may see STP bridge traffic but that is all.

In a corporate environment this can result in many calls to support and is essentially downtime and an unnecessary waste of resources.

Solution - Disable any Power Management and Wake on LAN Features in Windows

Generally these are any of the "Wake on LAN" or "Wake on Magic" or simiilarly named features.  Make sure you disable those under Device Manager for any of your NICs.

Quick Fix - If you have no control over the Windows NIC

The quick fix for this (if you don't have control over Windows) is powering off the machine and powering back.

In our example below, we create 2 hunt groups.

You could assume the #1 group is sales and #2 group is support etc.. and you can create more as needed.

The main part of hunt is the "list" option where you add each phone number that is to be part of the group

ephone-hunt 1 sequential
 pilot 2001
 list 1234
 timeout 10

ephone-hunt 2 sequential
 pilot 2002
 list 5678
 timeout 10
 

ephone-hunt 1 sequential
This sets up the first hunt group. The keyword sequential specifies that calls will be sent to each phone in the list one by one until someone answers, or the list is exhausted.

pilot 2001: This is the number used to reach the hunt group. When someone calls this number (2001), the system will follow the hunt sequence.

list 1234: This is the list of ephone-dn (directory numbers) the call will be routed to. In this case, the call will be sequentially forwarded to phones associated with extension 1234.

timeout 10: This is the amount of time (in seconds) the system will wait before trying the next phone in the sequence. In this case, it will wait for 10 seconds before forwarding the call to the next phone.

Step 01 - Download

Visit https://gns3.com/software/download

Click on "Linux" on the side under "Installation".

Run these commands:

What we are doing is adding the repo/ppa for gns3, then updating apt and then install gns3 and the gui.

sudo add-apt-repository ppa:gns3/ppa
sudo apt update                                
sudo apt install gns3-gui gns3-server

Run gns3 from the terminal by typing "gns3"

In our example we take "sound.mp3" and convert it to .wav.

Generally Asterisk for its wave needs one audio channel (-ac 1) / mono and 8000hz (-ar 8000) instead of the standard CD/MP3 of 44100hz.

Here is the command to convert into Asterisk .wav format:

 ffmpeg -i sound.mp3 -ac 1 -ar 8000 sound.wav

Errors Asterisk may give you if the format is wrong:

    -- Executing [91781891@cme:3] Playback("SIP/234-00000008", "/var/lib/asterisk/sounds/sound") in new stack
[Sep 26 15:45:32] WARNING[4400][C-00000009]: format_wav.c:111 check_header_fmt: Unexpected frequency mismatch 44100 (expecting 8000)
[Sep 26 15:45:32] WARNING[4400][C-00000009]: file.c:510 fn_wrapper: Unable to open format wav
[Sep 26 15:45:32] WARNING[4400][C-00000009]: file.c:1303 ast_streamfile: Unable to open /var/lib/asterisk/sounds/sound (format (ulaw)): No such file or directory
[Sep 26 15:45:32] WARNING[4400][C-00000009]: app_playback.c:513 playback_exec: Playback failed on SIP/234-00000008 for /var/lib/asterisk/sounds/sound
 

    -- Executing [9132131@cme:3] Playback("SIP/234-00000006", "/var/lib/asterisk/sounds/sound") in new stack
[Sep 26 15:44:18] WARNING[4384][C-00000007]: format_wav.c:102 check_header_fmt: Not in mono 2
[Sep 26 15:44:18] WARNING[4384][C-00000007]: file.c:510 fn_wrapper: Unable to open format wav
[Sep 26 15:44:18] WARNING[4384][C-00000007]: file.c:1303 ast_streamfile: Unable to open /var/lib/asterisk/sounds/sound (format (ulaw)): No such file or directory
[Sep 26 15:44:18] WARNING[4384][C-00000007]: app_playback.c:513 playback_exec: Playback failed on SIP/234-00000006 for /var/lib/asterisk/sounds/sound
 

#Remember that you need a valid gateway IP unless the Asterisk server is on the same subnet and LAN

Set Valid Gateway IP (if you don't have one already)

ip route 0.0.0.0 0.0.0.0 GATEWAYIP

Enable VOIP Trust

voice service voip
 ip address trusted list
  ipv4 0.0.0.0 0.0.0.0
  sip


Set Credentials For Asterisk and Register To Asterisk

sip-ua
 credentials username username password password realm asterisk
 registrar ipv4:ASTERISKIP expires 3600
 sip-server ipv4:ASTERISKIP
 

Set a Dial-Peer which uses Asterisk

This is so we can actually make calls out through it.  In our case we are simulating a typical office setup where you dial 9 for outside access.  This is a very "unoptimized" way that takes too long, in reality you probably want dial-peers that have more specific matches eg for local numbers (713-403-1234) format.

dial-peer voice 123 voip
 destination-pattern 9T
 session protocol sipv2
 session target ipv4:ASTERISKIP
 codec g711ulaw


At this point if you have done things correctly, all calls beginning with a 9 will go out through Asterisk (unless you already have more specific dial-peers).

Remember Asterisk must be configured to accept the client.  In some cases, you may need to modify the security users for the "friend" on Asterisk.

This is how we configure outside PSTN access or dialing through another SIP trunk.  Beware that this a simplistic example that neglects most security including SRTP

First you'll need to be in config mode:

Step - 1 Enter Voice Service VOIP security options

There are more options but for now we'll just focus on security/allowing connections to and from our phones and the trunk.

Router(config)#voice service voip
 

Then enter the trusted IP list:

Router(conf-voi-serv)#ip address trusted list

Set Trusted IPs:

In practice you would set the static IPs/ranges of your SIP trunk(s)

o be easy/insecure for testing just set any IP as being trusted (for trusted/LAN/testing only!)

Router(cfg-iptrust-list)#ipv4 0.0.0.0 0.0.0.0


To set a specify IP

This could be one of your SIP trunk IPs

Router(cfg-iptrust-list)#ipv4 192.168.5.20 255.255.255.0

To set a range (/24):

This may be a range of phones on your internal network that you want to trust.

Router(cfg-iptrust-list)#ipv4 10.0.0.0 255.255.255.0

You can add more entries as needed.

Step 2 - Setup Trunk

We'll set our preferred codec list

Router(config)#voice class codec 1

Router(config-class)#codec preference 1 g711ulaw
Router(config-class)#codec preference 2 g729r8

Go back to config mode:

Here is how we setup our trunk, depending on your service provider it may have different requirements (most providers will provide example configs).

Basically we are just authenticating using our username and password from our SIP provider and the last part "realm" is the IP or domain of the server.

authentication command will use the default username/password when authenticating. 

credentials command is really the same but you can have multiple of these (eg. if you have accounts with multiple SIP trunks/providers).

Router(config)#sip-ua

Router(config-sip-ua)#credentials username YOUR-USERNAME password 0 YOUR-PASSWORD realm IP-or-domain-of-Trunk.com

There is another option for credentials which can specify a specific phone number tied to the account (some providers do this and some may not).


Router(config-sip-ua)#credentials number 1234 username YOUR-USERNAME password 0 YOUR-PASSWORD realm IP-or-domain-of-Trunk.com

Router(config-sip-ua)#authentication username YOUR-USERNAME password 0 YOUR-PASSWORD realm IP-or-domain-of-Trunk.com

Step 3 - Use Trunk for outgoing calls

This is a generic use the trunk for all non-local calls, but in practice, you may have many providers and may have a more specific dial pattern so perhaps calls for country ABC go through dialpeer 4444 to give better or cheaper calls to a specific destination.

Be in config mode

In our example we call our dial-peer "123" but it could be any unused number.

Create the dial-peer

Router(config)#dial-peer voice 123 voip


We set destination pattern as anything starting with a 9 (the idea is that 9 is for dialing out, change according to your needs).

Router(config-dial-peer)#destination-pattern 9T


We set the protocol as sip, since we use this protocol for a SIP trunk of course.

Router(config-dial-peer)#session protocol sipv2


We set the codec, a very standard g711ulaw, but change according to your needs (what your provider also must support).

Router(config-dial-peer)#codec g711ulaw

Set callerid (usually should be the number your provider assigned):

In this example we set callerid as "12345678"

Router(config-dial-peer)#clid network-number 12345678

Set your session target

It's Cisco talk about which IP/domain do we send the calls to when this dial-peer is matched?

You can use a hostname like this

Router(config-dial-peer)#session target dns:your-server-domain

Or preferably if they have a static IP use this:

Router(config-dial-peer)#session target ipv4:192.168.5.254

Congrats, after this you should be able to make outgoing calls to the PSTN!

In newer versions this is a very stubborn issue.  Here is how you fix it.

Step 1 - Create networks.conf

sudo mkdir /etc/vbox/

sudo vi /etc/vbox/networks.conf

put this in:

In our case we can use the slash /16 range of 192.168.0..0, change the subnet according to your needs

* 192.168.0.0/16

Step 2 - Restart and close Virtualbox

service virtualbox restart

now reopen the GUI.

Edit your host-only network, you will find the IP changes back.  However, if you close and open Vbox you should see that it did indeed set.

Since newer versions of Ubuntu like 20, you will find there is no longer dynagen and that the dynamips provided is faulty and will segfault each time:

Cisco Router Simulation Platform (version 0.2.14-amd64/Linux stable)
Copyright (c) 2005-2011 Christophe Fillot.
Build date: Apr  3 2018 12:20:29

Local UUID: 3c1c0b7f-2fab-4fda-b40b-74841d1bcfe0

Instance ID set to 1.
netio_tap_create: unable to open TAP device tap1 (No such file or directory)
C7200 'default': unable to create NETIO descriptor for slot 0
IOS image file: c7200-adventerprisek9-mz.151-4.M.bin

ILT: loaded table "mips64j" from cache.
ILT: loaded table "mips64e" from cache.
ILT: loaded table "ppc32j" from cache.
ILT: loaded table "ppc32e" from cache.
CPU0: carved JIT exec zone of 64 Mb into 2048 pages of 32 Kb.
C7200 instance 'default' (id 1):
  VM Status  : 0
  RAM size   : 256 Mb
  IOMEM size : 0 Mb
  NVRAM size : 128 Kb
  NPE model  : npe-400
  Midplane   : vxr
  IOS image  : c7200-adventerprisek9-mz.151-4.M.bin

Loading ELF file 'c7200-adventerprisek9-mz.151-4.M.bin'...
ELF entry point: 0x80008000

C7200 'default': starting simulation (CPU0 PC=0xffffffffbfc00000), JIT enabled.

Segmentation fault (core dumped)
 

Solution - Compile The Newer Version of dynamips

As you can see from the output, the newest version in the repo is 0.2.14 which was built in 2018.  The versioning numbers themselves, without being a whole number, are understandly "alpha" sounding versions.

Clearly, something has to be fixed and there is no newer version in the repos, so we'll have to compile our own.

Fortunately the project was picked up and lives on here: https://github.com/GNS3/dynamips

Step 1 - Get Required Packages

apt install git libpcap0.8-dev libelf-dev cmake


Step 2 - Get Source Code:

git clone https://github.com/GNS3/dynamips

Cloning into 'dynamips'...
remote: Enumerating objects: 4195, done.
remote: Counting objects: 100% (775/775), done.
remote: Compressing objects: 100% (134/134), done.
remote: Total 4195 (delta 694), reused 641 (delta 641), pack-reused 3420 (from 1)
Receiving objects: 100% (4195/4195), 2.63 MiB | 20.39 MiB/s, done.
Resolving deltas: 100% (3019/3019), done.
 

Step 3 - Prepare Environment and Compile

cd dynamips


mkdir build


cd build


cmake .. -DDYNAMIPS_CODE=stable

You should see this at the end of successful compilation:

-- DYNAMIPS_FLAGS=-m64;-Wall;-O2;-fomit-frame-pointer
-- DYNAMIPS_DEFINITIONS=-DHAS_POSIX_MEMALIGN=1;-DDYNAMIPS_VERSION="0.2.23";-DJIT_ARCH="amd64";-DJIT_CPU=CPU_amd64;-DMIPS64_ARCH_INC_FILE="mips64_amd64_trans.h";-DPPC32_ARCH_INC_FILE="ppc32_amd64_trans.h";-D_FILE_OFFSET_BITS=64;-D_LARGEFILE_SOURCE;-D_LARGEFILE64_SOURCE;-DLINUX_ETH;-DGEN_ETH;-DHAS_RFC2553=1;-DOSNAME=Linux
-- DYNAMIPS_INCLUDES=/usr/include
-- DYNAMIPS_LIBRARIES=dl;rt;nsl;/usr/lib/x86_64-linux-gnu/libelf.so;-lpthread;/usr/lib/x86_64-linux-gnu/libpcap.so
-- configure - END
Summary:
  CMAKE_INSTALL_PREFIX               : /usr/local
  DYNAMIPS_ARCH                      : amd64
  DYNAMIPS_CODE                      : stable
  DYNAMIPS_RENAME                    : dynamips_amd64_stable -> dynamips  (auto)
  BUILD_NVRAM_EXPORT                 : ON
  BUILD_UDP_SEND                     : OFF
  BUILD_UDP_RECV                     : OFF
  Large File support                 : ENABLE_LARGEFILE=ON
  Linux Ethernet (RAW sockets)       : ENABLE_LINUX_ETH=ON  (linux_eth)
  Generic Ethernet (libpcap/WinPcap) : ENABLE_GEN_ETH=ON  (gen_eth)
  IPv6 support (RFC 2553)            : ENABLE_IPV6=ON
-- Configuring done
-- Generating done
-- Build files have been written to: /root/dynamips/build
root@d038b4fbf6e1:~/dynamips/build#
 

Now make and install

make


make install

ln -s /usr/local/bin/dynamips /bin

Confirm it is the right version:

You should as below that it is version 0.2.23 and the compilation date should be the date and time you did this.

dynamips|head
Please specify an IOS image filename
Cisco Router Simulation Platform (version 0.2.23-amd64/Linux stable)
Copyright (c) 2005-2011 Christophe Fillot.
Build date: Sep 16 2024 21:09:39
 

Make sure this makes sense for you but I've started to block a lot of commercial Cloud services and easily accessible providers as they are a very high source of abusive traffic.  The cost savings for a lot of organizations are huge, as you now have less bandwidth usage and less resource usage from garbage/bot/malicious traffic.  This mainly works for when you can be reasonably sure that your audience has no business visiting your service(s) from freely accessible commercial IP space (eg. Amazon, Huawei Cloud, Oracle Cloud etc..).  In general, any large service that offers Cloud to anyone without proper authentication, for cheap or offer any free trials/refunds (having a credit card is not enough authentication) is going to be a huge source of abusive traffic.

Some is not meant to be abusive but many of the SEO tools and services will scan your site, often causing a huge load by constantly hitting your services every second or several times per second.  Even if this does not slow your services down, there is a cost to it and you may be surprised at how much it is costing you or how much extra CPU, RAM and disk IO is wasted to abuse.

This will serve as a comprehensive list of suggested ranges to block.  If this causes issues from legitimate traffic, make sure you have a mechanism in place to whitelist "known good IPs" that you, your work or clients may be using.  Also discourage the use of rotating IPs unless there is a good reason, all traffic should be received by static IPs that are whitelisted.

If you have publicly accessible services, there are generally not too many good reasons why users should be hitting you from commercial cloud services or VPN providers (unless your users genuinely need a VPN for security or to access your services).  These services are a huge waste of resources as in many cases, I've seen 80-90% of traffic is just garbage traffic, and often it is a competitor or bad actor using these services to actively attack public services.

Now if you happen to know a service that uses a cloud service, such as Pinterest using Amazon and don't want to block it, you can whitelist their IP ranges as most reputable services will list their range of IPs for this purpose.

Note that this is not recommended as replacement for good security and firewalls but it already reduces the abusive traffic angle.  

However, given that the Cloud providers offer services to anyone, they are still a source of intentionally maliicious traffic because throw away accounts are allowed on many Cloud services (eg. free trial that one uses with a prepaid credit card for the purposes of abuse).

The presence or lack of presence of a Cloud provider here does not mean they are better or worse for abuse, but I just haven't gotten a chance to list them here yet.

All proposed blocks are based on real-world examples (eg. abusive traffic coming from those hosts).

In another post we'll list a bunch of bad user agents to block.

When not to use Cloud Providers

1.) If you value privacy or control of your data

2.) If you are using it as a VPN or other outward facing service there is a good chance it may be transparently blocked.

3.) If you must use a VPN, it is best to run it from one of your office's legitimate internet connection as these IPs are the least likely to be blocked.

Where are they really from?

Beware that user agents can be forged, many DDOS services will try to masquerade as a legitimate service or scanning tool but you will often find that there is no relation (eg. a bot hitting your site that has the Google bot agent set but is really from Digital Ocean and not truly Google).

Other proposed blocks:

In general any cheap provider and any large provider should probably be blocked.  This especially means providers that offer a free trial or X-day money back guarantee.

• GCP (Google Cloud)
• Vultr (Cloud)
• Hetzner (Cheap host)
• Microsoft Azure
• Oracle Cloud
• Worldstream (smaller but abusive traffic)
• Scaleway
• OVH
• Leaseweb
• IPXO LLC
• Server Destroyers LLC

Other Crap:

turnitin

It's a service for plagiarism but we've seen it hitting client servers abusively so it's the same as a DDOS/waste of resources/botnet since it offers no value to the hoster.

199.47.80.0/21

Block SEO Tools:

Popular tools include:

• Moz
• Ahrefs
• SEMRush
• Majestic SEO

If you aren't using the tools for the particular site(s) running then these should be blocked or at least give them a 403 based on the user agent.

We've seen several SEO tools used at once by determined attacker as a similar impact as DDOS by having reports run on the target site/URL done by several SEO tools or other scanning site performance tools that were otherwise legitimate.

Block Search Engines/Crawlers:

If you don't get traffic from certain search engines then consider blocking them.  There are dozens of other smaller crawling services that cause huge issues.

Block AI Tools:

This has the double effect of preserving your resources and stopping AI from stealing or using your info in their replies.

Our experience is that most providers simply don't have time or care about the large amount of abuse reports they get (eg. complain and you will usually never receive any response or action).

Huawei Cloud is a good idea to block as I've seen too many cases of abuse from their ranges despite them not being considered such a huge provider compared to AWS.

1.178.32.0/20
1.178.48.0/20
101.44.0.0/20
101.44.144.0/20
101.44.16.0/20
101.44.160.0/20
101.44.173.0/24
101.44.176.0/20
101.44.192.0/20
101.44.208.0/22
101.44.212.0/22
101.44.216.0/22
101.44.220.0/22
101.44.224.0/22
101.44.228.0/22
101.44.232.0/22
101.44.236.0/22
101.44.244.0/22
101.44.248.0/22
101.44.252.0/24
101.44.253.0/24
101.44.254.0/24
101.44.255.0/24
101.44.32.0/20
101.44.48.0/20
101.44.64.0/20
101.44.80.0/20
101.44.96.0/20
101.46.0.0/20
101.46.128.0/21
101.46.136.0/21
101.46.144.0/21
101.46.152.0/21
101.46.160.0/21
101.46.168.0/21
101.46.176.0/21
101.46.184.0/21
101.46.192.0/21
101.46.208.0/21
101.46.216.0/21
101.46.236.0/22
101.46.240.0/22
101.46.248.0/22
101.46.252.0/24
101.46.254.0/24
101.46.255.0/24
101.46.32.0/20
101.46.48.0/20
101.46.64.0/20
103.215.0.0/24
103.215.1.0/24
103.215.3.0/24
103.239.72.0/24
103.240.157.0/24
103.255.61.0/24
103.255.62.0/24
110.238.100.0/22
110.238.104.0/21
110.238.112.0/21
110.238.120.0/22
110.238.124.0/22
110.238.64.0/21
110.238.72.0/21
110.238.80.0/20
110.238.96.0/24
110.238.98.0/24
110.238.99.0/24
110.239.127.0/24
110.239.64.0/19
110.239.96.0/19
110.41.210.0/24
110.41.90.0/24
111.119.192.0/20
111.119.208.0/20
111.119.224.0/20
111.119.240.0/20
114.119.128.0/19
114.119.160.0/21
114.119.168.0/24
114.119.169.0/24
114.119.170.0/24
114.119.171.0/24
114.119.172.0/22
114.119.176.0/20
115.30.32.0/20
115.30.48.0/20
119.12.160.0/20
119.13.112.0/20
119.13.160.0/24
119.13.161.0/24
119.13.162.0/23
119.13.163.0/24
119.13.164.0/22
119.13.168.0/24
119.13.169.0/24
119.13.170.0/24
119.13.171.0/24
119.13.172.0/24
119.13.173.0/24
119.13.174.0/24
119.13.175.0/24
119.13.64.0/24
119.13.65.0/24
119.13.66.0/23
119.13.68.0/22
119.13.72.0/22
119.13.76.0/22
119.13.80.0/21
119.13.88.0/22
119.13.92.0/22
119.13.96.0/20
119.8.0.0/21
119.8.128.0/24
119.8.129.0/24
119.8.13.0/24
119.8.130.0/23
119.8.132.0/22
119.8.136.0/21
119.8.144.0/20
119.8.15.0/24
119.8.160.0/19
119.8.18.0/24
119.8.192.0/20
119.8.192.0/21
119.8.200.0/21
119.8.208.0/20
119.8.21.0/24
119.8.22.0/24
119.8.224.0/24
119.8.227.0/24
119.8.228.0/22
119.8.23.0/24
119.8.232.0/21
119.8.24.0/21
119.8.240.0/23
119.8.242.0/23
119.8.244.0/24
119.8.245.0/24
119.8.246.0/24
119.8.247.0/24
119.8.248.0/24
119.8.249.0/24
119.8.250.0/24
119.8.252.0/24
119.8.253.0/24
119.8.254.0/23
119.8.32.0/19
119.8.4.0/24
119.8.64.0/22
119.8.68.0/24
119.8.69.0/24
119.8.70.0/24
119.8.71.0/24
119.8.72.0/21
119.8.8.0/21
119.8.80.0/20
119.8.96.0/19
121.91.152.0/21
121.91.168.0/21
121.91.200.0/21
121.91.200.0/24
122.8.128.0/20
122.8.144.0/20
122.8.160.0/20
122.8.176.0/21
122.8.184.0/22
122.8.188.0/22
124.243.128.0/18
124.243.156.0/24
124.243.157.0/24
124.243.158.0/24
124.243.159.0/24
124.71.248.0/24
124.71.249.0/24
124.71.252.0/24
124.71.253.0/24
124.81.0.0/20
124.81.16.0/20
124.81.160.0/20
124.81.176.0/20
124.81.192.0/20
124.81.208.0/20
124.81.224.0/20
124.81.240.0/20
124.81.32.0/20
124.81.48.0/20
124.81.64.0/20
124.81.80.0/20
14.137.132.0/22
14.137.136.0/22
14.137.140.0/22
14.137.152.0/24
14.137.153.0/24
14.137.154.0/24
14.137.156.0/24
14.137.157.0/24
14.137.161.0/24
14.137.169.0/24
14.137.170.0/23
14.137.172.0/22
149.232.128.0/20
149.232.144.0/20
150.40.128.0/20
150.40.144.0/20
150.40.160.0/20
150.40.176.0/20
150.40.192.0/20
150.40.208.0/20
150.40.224.0/20
150.40.240.0/20
154.220.192.0/19
154.81.16.0/20
154.83.0.0/23
154.86.32.0/20
154.86.48.0/20
154.93.100.0/23
154.93.104.0/23
156.227.22.0/23
156.230.32.0/21
156.230.40.0/21
156.230.64.0/18
156.232.16.0/20
156.240.128.0/18
156.249.32.0/20
156.253.16.0/20
159.138.0.0/20
159.138.112.0/21
159.138.114.0/24
159.138.120.0/22
159.138.124.0/24
159.138.125.0/24
159.138.126.0/23
159.138.128.0/20
159.138.144.0/20
159.138.152.0/21
159.138.16.0/22
159.138.160.0/20
159.138.176.0/23
159.138.178.0/24
159.138.179.0/24
159.138.180.0/24
159.138.181.0/24
159.138.182.0/23
159.138.188.0/23
159.138.190.0/23
159.138.192.0/20
159.138.20.0/22
159.138.208.0/21
159.138.216.0/22
159.138.220.0/23
159.138.224.0/20
159.138.24.0/21
159.138.240.0/20
159.138.32.0/20
159.138.48.0/20
159.138.64.0/21
159.138.67.0/24
159.138.76.0/24
159.138.77.0/24
159.138.78.0/24
159.138.79.0/24
159.138.80.0/20
159.138.96.0/20
166.108.192.0/20
166.108.208.0/20
166.108.224.0/20
166.108.240.0/20
176.52.128.0/20
176.52.144.0/20
180.87.192.0/20
180.87.208.0/20
180.87.224.0/20
180.87.240.0/20
182.160.0.0/20
182.160.16.0/24
182.160.17.0/24
182.160.18.0/23
182.160.20.0/24
182.160.24.0/21
182.160.36.0/22
182.160.49.0/24
182.160.52.0/22
182.160.56.0/24
182.160.57.0/24
182.160.58.0/24
182.160.59.0/24
182.160.60.0/24
182.160.61.0/24
182.160.62.0/24
183.87.112.0/20
183.87.128.0/20
183.87.144.0/20
183.87.32.0/20
183.87.48.0/20
183.87.64.0/20
183.87.80.0/20
183.87.96.0/20
188.239.0.0/20
188.239.16.0/20
189.1.192.0/20
189.1.208.0/20
189.1.224.0/20
189.1.240.0/20
189.28.112.0/20
189.28.96.0/20
190.92.192.0/19
190.92.224.0/19
190.92.248.0/24
190.92.252.0/24
190.92.253.0/24
190.92.254.0/24
193.105.244.0/23
193.84.248.0/23
201.77.32.0/20
202.170.88.0/21
203.123.80.0/20
203.167.20.0/23
203.167.22.0/24
212.34.192.0/20
212.34.208.0/20
212.34.216.32/32
213.250.128.0/20
213.250.144.0/20
27.106.0.0/20
27.106.112.0/20
27.106.16.0/20
27.106.32.0/20
27.106.48.0/20
27.106.64.0/20
27.106.80.0/20
27.106.96.0/20
43.225.140.0/22
43.255.104.0/22
45.194.104.0/21
45.199.144.0/22
45.202.128.0/19
45.202.160.0/20
45.202.176.0/21
45.202.184.0/21
45.203.32.0/21
45.203.40.0/21
46.250.160.0/20
46.250.176.0/20
49.0.192.0/21
49.0.200.0/21
49.0.224.0/22
49.0.228.0/22
49.0.232.0/21
49.0.240.0/20
80.238.132.0/22
80.238.136.0/22
80.238.180.0/24
80.238.190.0/24
80.238.192.0/20
80.238.208.0/20
80.238.224.0/20
83.101.0.0/21
83.101.16.0/21
83.101.48.0/21
83.101.56.0/21
87.119.12.0/24
94.45.160.0/24
94.45.161.0/24
94.45.163.0/24
94.74.112.0/21
94.74.120.0/21
94.74.64.0/20
94.74.80.0/20
94.74.96.0/20

Digital Ocean

Digital Ocean is a huge offender based on experience and we have never seen any response to abuse complaints.

5.101.96.0/21
5.101.104.0/22
5.101.108.0/24
5.101.109.0/24
5.101.110.0/24
5.101.111.0/24
24.144.64.0/22
24.144.68.0/22
24.144.72.0/24
24.144.73.0/24
24.144.74.0/23
24.144.76.0/22
24.144.80.0/20
24.144.96.0/19
24.199.64.0/22
24.199.68.0/22
24.199.72.0/21
24.199.80.0/20
24.199.96.0/20
24.199.112.0/20
37.139.0.0/19
45.55.0.0/19
45.55.32.0/19
45.55.64.0/19
45.55.96.0/22
45.55.100.0/22
45.55.104.0/22
45.55.108.0/22
45.55.112.0/22
45.55.116.0/22
45.55.120.0/22
45.55.124.0/22
45.55.128.0/18
45.55.192.0/18
46.101.0.0/18
46.101.64.0/22
46.101.68.0/22
46.101.72.0/21
46.101.80.0/20
46.101.96.0/19
46.101.128.0/17
64.23.128.0/20
64.23.144.0/20
64.23.160.0/20
64.23.176.0/20
64.23.192.0/19
64.23.224.0/20
64.23.240.0/20
64.225.0.0/20
64.225.16.0/20
64.225.32.0/20
64.225.48.0/20
64.225.64.0/20
64.225.80.0/22
64.225.84.0/22
64.225.88.0/22
64.225.92.0/22
64.225.96.0/20
64.225.112.0/20
64.226.64.0/20
64.226.80.0/20
64.226.96.0/20
64.226.112.0/20
64.227.0.0/20
64.227.16.0/20
64.227.32.0/20
64.227.48.0/20
64.227.64.0/20
64.227.80.0/20
64.227.96.0/20
64.227.112.0/20
64.227.128.0/19
64.227.160.0/20
64.227.176.0/20
67.205.128.0/20
67.205.144.0/20
67.205.160.0/20
67.205.176.0/20
67.207.64.0/23
67.207.66.0/24
67.207.68.0/22
67.207.72.0/22
67.207.76.0/22
67.207.80.0/20
68.183.0.0/20
68.183.16.0/20
68.183.32.0/20
68.183.48.0/20
68.183.64.0/20
68.183.80.0/20
68.183.96.0/20
68.183.112.0/20
68.183.128.0/20
68.183.144.0/20
68.183.160.0/20
68.183.176.0/20
68.183.192.0/20
68.183.208.0/20
68.183.224.0/20
68.183.240.0/22
68.183.244.0/22
68.183.248.0/22
68.183.252.0/22
69.55.48.0/22
69.55.49.0/24
69.55.54.0/24
69.55.55.0/24
69.55.58.0/23
69.55.60.0/22
80.240.128.0/20
82.196.0.0/20
95.85.0.0/18
103.253.144.0/22
104.131.0.0/18
104.131.64.0/18
104.131.128.0/20
104.131.144.0/20
104.131.160.0/20
104.131.176.0/20
104.131.192.0/19
104.131.224.0/19
104.236.0.0/18
104.236.64.0/18
104.236.128.0/18
104.236.192.0/18
104.248.0.0/20
104.248.16.0/20
104.248.32.0/20
104.248.48.0/20
104.248.64.0/20
104.248.80.0/20
104.248.96.0/22
104.248.100.0/22
104.248.104.0/22
104.248.108.0/22
104.248.112.0/20
104.248.128.0/20
104.248.144.0/20
104.248.160.0/20
104.248.176.0/20
104.248.192.0/20
104.248.208.0/20
104.248.224.0/20
104.248.240.0/20
107.170.0.0/17
107.170.128.0/19
107.170.160.0/19
107.170.192.0/18
128.199.0.0/20
128.199.16.0/20
128.199.32.0/19
128.199.64.0/18
128.199.128.0/18
128.199.192.0/18
134.122.0.0/20
134.122.16.0/20
134.122.32.0/20
134.122.48.0/20
134.122.64.0/20
134.122.80.0/20
134.122.96.0/20
134.122.112.0/20
134.209.0.0/20
134.209.16.0/20
134.209.32.0/20
134.209.48.0/20
134.209.64.0/20
134.209.80.0/20
134.209.96.0/20
134.209.112.0/20
134.209.128.0/22
134.209.132.0/22
134.209.136.0/22
134.209.140.0/22
134.209.144.0/20
134.209.160.0/20
134.209.176.0/20
134.209.192.0/20
134.209.208.0/20
134.209.224.0/20
134.209.240.0/20
137.184.0.0/20
137.184.16.0/20
137.184.32.0/20
137.184.48.0/20
137.184.64.0/20
137.184.80.0/20
137.184.96.0/20
137.184.112.0/20
137.184.128.0/20
137.184.144.0/20
137.184.160.0/20
137.184.176.0/20
137.184.192.0/20
137.184.208.0/20
137.184.224.0/20
137.184.240.0/22
137.184.244.0/22
137.184.248.0/22
137.184.252.0/24
137.184.254.0/24
137.184.255.0/24
138.68.0.0/20
138.68.16.0/20
138.68.32.0/24
138.68.34.0/24
138.68.36.0/22
138.68.40.0/21
138.68.48.0/20
138.68.64.0/20
138.68.80.0/20
138.68.96.0/20
138.68.112.0/22
138.68.116.0/22
138.68.120.0/23
138.68.122.0/23
138.68.124.0/22
138.68.128.0/20
138.68.144.0/20
138.68.160.0/20
138.68.176.0/20
138.68.192.0/22
138.68.196.0/22
138.68.200.0/22
138.68.204.0/22
138.68.208.0/20
138.68.224.0/20
138.68.240.0/20
138.197.0.0/20
138.197.16.0/20
138.197.32.0/20
138.197.48.0/22
138.197.52.0/22
138.197.56.0/22
138.197.60.0/22
138.197.64.0/20
138.197.80.0/20
138.197.96.0/20
138.197.112.0/20
138.197.128.0/20
138.197.144.0/20
138.197.160.0/20
138.197.176.0/20
138.197.192.0/20
138.197.208.0/20
138.197.224.0/22
138.197.228.0/22
138.197.232.0/22
138.197.236.0/22
138.197.240.0/22
138.197.252.0/22
139.59.0.0/20
139.59.16.0/20
139.59.32.0/20
139.59.48.0/22
139.59.52.0/22
139.59.56.0/21
139.59.64.0/20
139.59.80.0/20
139.59.96.0/20
139.59.112.0/20
139.59.128.0/19
139.59.160.0/20
139.59.176.0/20
139.59.192.0/22
139.59.196.0/22
139.59.200.0/22
139.59.204.0/22
139.59.208.0/21
139.59.216.0/22
139.59.220.0/22
139.59.224.0/20
139.59.240.0/20
141.0.169.0/24
141.0.170.0/24
142.93.0.0/20
142.93.16.0/20
142.93.32.0/20
142.93.48.0/20
142.93.64.0/20
142.93.80.0/20
142.93.96.0/20
142.93.112.0/20
142.93.128.0/20
142.93.144.0/20
142.93.160.0/20
142.93.176.0/20
142.93.192.0/20
142.93.208.0/20
142.93.224.0/20
142.93.240.0/20
143.110.128.0/20
143.110.144.0/20
143.110.160.0/20
143.110.176.0/20
143.110.192.0/20
143.110.208.0/20
143.110.224.0/20
143.110.240.0/20
143.198.0.0/20
143.198.16.0/20
143.198.32.0/20
143.198.48.0/20
143.198.64.0/20
143.198.80.0/20
143.198.96.0/20
143.198.112.0/20
143.198.128.0/20
143.198.144.0/20
143.198.160.0/20
143.198.176.0/20
143.198.192.0/20
143.198.208.0/20
143.198.224.0/20
143.198.240.0/22
143.198.244.0/22
143.198.248.0/22
143.244.128.0/20
143.244.144.0/20
143.244.160.0/20
143.244.176.0/20
143.244.196.0/22
143.244.200.0/22
143.244.204.0/22
143.244.208.0/22
143.244.212.0/22
143.244.217.0/24
143.244.218.0/24
143.244.219.0/24
143.244.220.0/22
144.126.192.0/20
144.126.208.0/20
144.126.224.0/20
144.126.240.0/22
144.126.244.0/22
144.126.248.0/22
144.126.252.0/22
146.185.128.0/19
146.185.160.0/20
146.185.176.0/21
146.185.184.0/21
146.190.0.0/22
146.190.4.0/22
146.190.8.0/22
146.190.12.0/22
146.190.16.0/20
146.190.32.0/19
146.190.64.0/20
146.190.80.0/20
146.190.96.0/20
146.190.112.0/20
146.190.128.0/19
146.190.160.0/20
146.190.176.0/22
146.190.184.0/22
146.190.188.0/22
146.190.192.0/22
146.190.196.0/22
146.190.200.0/22
146.190.204.0/22
146.190.208.0/20
146.190.224.0/20
146.190.240.0/20
147.182.128.0/20
147.182.144.0/20
147.182.160.0/20
147.182.176.0/20
147.182.192.0/20
147.182.208.0/20
147.182.224.0/20
147.182.240.0/20
152.42.128.0/20
152.42.144.0/22
152.42.148.0/22
152.42.152.0/22
152.42.156.0/22
152.42.160.0/19
152.42.192.0/19
152.42.224.0/20
152.42.240.0/20
157.230.0.0/20
157.230.16.0/20
157.230.32.0/20
157.230.48.0/20
157.230.64.0/22
157.230.68.0/22
157.230.72.0/22
157.230.76.0/22
157.230.80.0/20
157.230.96.0/20
157.230.112.0/20
157.230.128.0/20
157.230.144.0/20
157.230.160.0/20
157.230.176.0/20
157.230.192.0/22
157.230.196.0/22
157.230.200.0/22
157.230.204.0/22
157.230.208.0/20
157.230.224.0/20
157.230.240.0/20
157.245.0.0/20
157.245.16.0/22
157.245.20.0/22
157.245.24.0/22
157.245.28.0/22
157.245.32.0/20
157.245.48.0/20
157.245.64.0/20
157.245.80.0/20
157.245.96.0/20
157.245.112.0/20
157.245.128.0/20
157.245.144.0/20
157.245.160.0/20
157.245.176.0/20
157.245.192.0/20
157.245.208.0/20
157.245.224.0/20
157.245.240.0/20
159.65.0.0/20
159.65.16.0/20
159.65.32.0/20
159.65.48.0/20
159.65.64.0/20
159.65.80.0/20
159.65.96.0/20
159.65.112.0/20
159.65.128.0/20
159.65.144.0/20
159.65.160.0/20
159.65.176.0/20
159.65.192.0/20
159.65.208.0/22
159.65.212.0/22
159.65.216.0/21
159.65.224.0/20
159.65.240.0/20
159.89.0.0/20
159.89.16.0/20
159.89.32.0/20
159.89.48.0/21
159.89.58.0/24
159.89.59.0/24
159.89.60.0/24
159.89.61.0/24
159.89.62.0/24
159.89.63.0/24
159.89.64.0/20
159.89.80.0/20
159.89.96.0/20
159.89.112.0/20
159.89.128.0/20
159.89.144.0/20
159.89.160.0/20
159.89.176.0/20
159.89.192.0/20
159.89.208.0/22
159.89.212.0/22
159.89.216.0/22
159.89.220.0/22
159.89.224.0/20
159.89.240.0/22
159.89.244.0/22
159.89.248.0/22
159.89.252.0/22
159.203.0.0/20
159.203.16.0/20
159.203.32.0/20
159.203.48.0/22
159.203.52.0/22
159.203.56.0/21
159.203.64.0/20
159.203.80.0/20
159.203.96.0/20
159.203.112.0/20
159.203.128.0/20
159.203.144.0/22
159.203.148.0/22
159.203.152.0/22
159.203.156.0/22
159.203.160.0/20
159.203.176.0/20
159.203.192.0/20
159.203.208.0/20
159.203.224.0/20
159.203.240.0/20
159.223.0.0/20
159.223.16.0/20
159.223.32.0/20
159.223.48.0/20
159.223.64.0/20
159.223.80.0/20
159.223.96.0/20
159.223.112.0/20
159.223.128.0/20
159.223.144.0/20
159.223.160.0/19
159.223.192.0/20
159.223.208.0/20
159.223.224.0/20
159.223.240.0/22
159.223.244.0/22
159.223.248.0/22
161.35.0.0/20
161.35.16.0/20
161.35.32.0/20
161.35.48.0/20
161.35.64.0/20
161.35.80.0/20
161.35.96.0/20
161.35.112.0/20
161.35.128.0/20
161.35.144.0/20
161.35.160.0/20
161.35.176.0/20
161.35.192.0/20
161.35.208.0/20
161.35.224.0/20
161.35.240.0/22
161.35.244.0/22
161.35.248.0/22
161.35.252.0/22
162.243.0.0/17
162.243.128.0/19
162.243.160.0/20
162.243.184.0/22
162.243.188.0/23
162.243.190.0/24
162.243.191.0/24
162.243.192.0/18
163.47.8.0/22
164.90.128.0/20
164.90.144.0/20
164.90.160.0/20
164.90.176.0/20
164.90.192.0/20
164.90.208.0/20
164.90.224.0/20
164.90.240.0/22
164.90.244.0/22
164.90.248.0/24
164.90.249.0/24
164.90.250.0/24
164.90.252.0/22
164.92.64.0/19
164.92.96.0/19
164.92.128.0/20
164.92.144.0/20
164.92.160.0/20
164.92.176.0/20
164.92.192.0/20
164.92.208.0/20
164.92.224.0/20
164.92.240.0/20
165.22.0.0/20
165.22.16.0/20
165.22.32.0/20
165.22.48.0/20
165.22.64.0/20
165.22.80.0/20
165.22.96.0/20
165.22.112.0/20
165.22.128.0/20
165.22.144.0/20
165.22.160.0/20
165.22.176.0/20
165.22.192.0/20
165.22.208.0/20
165.22.224.0/20
165.22.240.0/20
165.227.0.0/20
165.227.16.0/20
165.227.32.0/20
165.227.48.0/20
165.227.64.0/20
165.227.80.0/20
165.227.96.0/20
165.227.112.0/20
165.227.128.0/20
165.227.144.0/20
165.227.160.0/20
165.227.176.0/20
165.227.192.0/20
165.227.208.0/20
165.227.224.0/20
165.227.240.0/22
165.227.244.0/22
165.227.248.0/22
165.227.252.0/22
165.232.32.0/20
165.232.48.0/20
165.232.64.0/20
165.232.80.0/20
165.232.96.0/20
165.232.112.0/20
165.232.128.0/20
165.232.144.0/20
165.232.160.0/20
165.232.176.0/20
167.71.0.0/20
167.71.16.0/20
167.71.32.0/20
167.71.48.0/20
167.71.64.0/20
167.71.80.0/20
167.71.96.0/20
167.71.112.0/20
167.71.128.0/20
167.71.144.0/20
167.71.160.0/20
167.71.176.0/20
167.71.192.0/20
167.71.208.0/20
167.71.224.0/20
167.71.240.0/20
167.99.0.0/20
167.99.16.0/22
167.99.20.0/22
167.99.24.0/22
167.99.28.0/22
167.99.32.0/20
167.99.48.0/20
167.99.64.0/20
167.99.80.0/20
167.99.96.0/20
167.99.112.0/20
167.99.128.0/20
167.99.144.0/20
167.99.160.0/20
167.99.176.0/20
167.99.192.0/20
167.99.208.0/20
167.99.224.0/20
167.99.240.0/20
167.172.0.0/22
167.172.4.0/22
167.172.8.0/22
167.172.12.0/22
167.172.16.0/20
167.172.32.0/20
167.172.48.0/20
167.172.64.0/20
167.172.80.0/20
167.172.96.0/20
167.172.112.0/20
167.172.128.0/20
167.172.144.0/20
167.172.160.0/20
167.172.176.0/20
167.172.192.0/20
167.172.208.0/20
167.172.224.0/20
167.172.240.0/20
170.64.128.0/18
170.64.192.0/19
170.64.224.0/20
170.64.240.0/21
170.64.248.0/21
174.138.0.0/20
174.138.16.0/20
174.138.32.0/20
174.138.48.0/20
174.138.64.0/20
174.138.80.0/20
174.138.96.0/22
174.138.100.0/22
174.138.104.0/22
174.138.108.0/22
174.138.112.0/22
174.138.116.0/22
174.138.120.0/22
174.138.124.0/22
178.62.0.0/18
178.62.64.0/18
178.62.128.0/18
178.62.192.0/18
178.128.0.0/20
178.128.16.0/20
178.128.32.0/20
178.128.48.0/20
178.128.64.0/20
178.128.80.0/20
178.128.96.0/20
178.128.112.0/20
178.128.128.0/22
178.128.132.0/22
178.128.136.0/22
178.128.140.0/22
178.128.144.0/20
178.128.160.0/20
178.128.176.0/20
178.128.192.0/20
178.128.208.0/20
178.128.224.0/20
178.128.240.0/20
185.14.184.0/22
188.166.0.0/18
188.166.64.0/18
188.166.128.0/22
188.166.132.0/22
188.166.136.0/22
188.166.140.0/22
188.166.144.0/20
188.166.160.0/21
188.166.168.0/21
188.166.176.0/20
188.166.192.0/22
188.166.196.0/22
188.166.200.0/22
188.166.204.0/22
188.166.208.0/20
188.166.224.0/20
188.166.240.0/20
188.226.128.0/17
192.34.56.0/21
192.81.208.0/21
192.81.216.0/22
192.81.220.0/22
192.241.128.0/19
192.241.160.0/19
192.241.192.0/19
192.241.224.0/20
192.241.240.0/20
198.199.64.0/20
198.199.80.0/21
198.199.88.0/22
198.199.92.0/22
198.199.96.0/20
198.199.112.0/21
198.199.120.0/22
198.199.124.0/22
198.211.96.0/20
198.211.112.0/22
198.211.116.0/23
198.211.118.0/23
198.211.120.0/21
204.48.16.0/20
206.81.0.0/20
206.81.16.0/20
206.189.0.0/20
206.189.16.0/20
206.189.32.0/20
206.189.48.0/20
206.189.64.0/20
206.189.80.0/20
206.189.96.0/20
206.189.112.0/20
206.189.128.0/20
206.189.144.0/20
206.189.160.0/20
206.189.176.0/20
206.189.192.0/20
206.189.208.0/20
206.189.224.0/20
206.189.240.0/22
206.189.244.0/22
206.189.248.0/22
206.189.252.0/22
207.154.192.0/20
207.154.208.0/20
207.154.224.0/20
207.154.240.0/20
208.68.36.0/22
209.38.0.0/22
209.38.4.0/22
209.38.8.0/21
209.38.16.0/20
209.38.32.0/20
209.38.48.0/22
209.38.52.0/22
209.38.64.0/20
209.38.96.0/20
209.38.128.0/19
209.38.160.0/22
209.38.164.0/22
209.38.168.0/22
209.38.172.0/22
209.38.176.0/20
209.38.192.0/19
209.38.224.0/19
209.97.128.0/20
209.97.144.0/20
209.97.160.0/20
209.97.176.0/20

Google Cloud (GCP):

https://www.gstatic.com/ipranges/cloud.json

 34.1.208.0/20
 34.35.0.0/16
 34.152.86.0/23
 34.177.50.0/23
 34.80.0.0/15
 34.137.0.0/16
 35.185.128.0/19
 35.185.160.0/20
 35.187.144.0/20
 35.189.160.0/19
 35.194.128.0/17
 35.201.128.0/17
 35.206.192.0/18
 35.220.32.0/21
 35.221.128.0/17
 35.229.128.0/17
 35.234.0.0/18
 35.235.16.0/20
 35.236.128.0/18
 35.242.32.0/21
 104.155.192.0/19
 104.155.224.0/20
 104.199.128.0/18
 104.199.192.0/19
 104.199.224.0/20
 104.199.242.0/23
 104.199.244.0/22
 104.199.248.0/21
 107.167.176.0/20
 130.211.240.0/20
 34.92.0.0/16
 34.96.128.0/17
 34.104.88.0/21
 34.124.24.0/21
 34.150.0.0/17
 35.215.128.0/18
 35.220.27.0/24
 35.220.128.0/17
 35.241.64.0/18
 35.242.27.0/24
 35.243.8.0/21
 34.84.0.0/16
 34.85.0.0/17
 34.104.62.0/23
 34.104.128.0/17
 34.127.190.0/23
 34.146.0.0/16
 34.157.64.0/20
 34.157.164.0/22
 34.157.192.0/20
 35.187.192.0/19
 35.189.128.0/19
 35.190.224.0/20
 35.194.96.0/19
 35.200.0.0/17
 35.213.0.0/17
 35.220.56.0/22
 35.221.64.0/18
 35.230.240.0/20
 35.242.56.0/22
 35.243.64.0/18
 104.198.80.0/20
 104.198.112.0/20
 34.97.0.0/16
 34.104.49.0/24
 34.127.177.0/24
 35.217.128.0/17
 35.220.45.0/24
 35.242.45.0/24
 35.243.56.0/21
 34.0.96.0/19
 34.22.64.0/19
 34.22.96.0/20
 34.47.64.0/18
 34.50.0.0/18
 34.64.32.0/19
 34.64.64.0/22
 34.64.68.0/22
 34.64.72.0/21
 34.64.80.0/20
 34.64.96.0/19
 34.64.128.0/22
 34.64.132.0/22
 34.64.136.0/21
 34.64.144.0/20
 34.64.160.0/19
 34.64.192.0/18
 35.216.0.0/17
 34.0.227.0/24
 34.47.128.0/17
 34.93.0.0/16
 34.100.128.0/17
 34.104.108.0/23
 34.124.44.0/23
 34.152.64.0/22
 34.153.58.0/23
 34.153.250.0/23
 34.157.87.0/24
 34.157.215.0/24
 34.177.32.0/22
 35.200.128.0/17
 35.201.41.0/24
 35.207.192.0/18
 35.220.42.0/24
 35.234.208.0/20
 35.242.42.0/24
 35.244.0.0/18
 34.0.0.0/20
 34.104.120.0/23
 34.124.56.0/23
 34.126.208.0/20
 34.131.0.0/16
 34.153.32.0/24
 34.153.224.0/24
 34.1.128.0/20
 34.1.192.0/20
 34.21.128.0/17
 34.87.0.0/17
 34.87.128.0/18
 34.104.58.0/23
 34.104.106.0/23
 34.124.42.0/23
 34.124.128.0/17
 34.126.64.0/18
 34.126.128.0/18
 34.128.44.0/23
 34.128.60.0/23
 34.142.128.0/17
 34.143.128.0/17
 34.153.40.0/23
 34.153.232.0/23
 34.157.82.0/23
 34.157.88.0/23
 34.157.210.0/23
 35.185.176.0/20
 35.186.144.0/20
 35.187.224.0/19
 35.197.128.0/19
 35.198.192.0/18
 35.213.128.0/18
 35.220.24.0/23
 35.234.192.0/20
 35.240.128.0/17
 35.242.24.0/23
 35.247.128.0/18
 34.34.216.0/21
 34.50.64.0/18
 34.101.18.0/24
 34.101.20.0/22
 34.101.24.0/22
 34.101.32.0/19
 34.101.64.0/18
 34.101.128.0/17
 34.128.64.0/18
 34.152.68.0/24
 34.153.44.0/24
 34.153.236.0/24
 34.157.254.0/24
 35.219.0.0/17
 34.40.128.0/17
 34.87.192.0/18
 34.104.104.0/23
 34.116.64.0/18
 34.124.40.0/23
 34.128.36.0/24
 34.128.48.0/24
 34.151.64.0/18
 34.151.128.0/18
 35.189.0.0/18
 35.197.160.0/19
 35.201.0.0/19
 35.213.192.0/18
 35.220.41.0/24
 35.234.224.0/20
 35.242.41.0/24
 35.244.64.0/18
 34.0.16.0/20
 34.1.176.0/20
 34.104.122.0/23
 34.124.58.0/23
 34.126.192.0/20
 34.129.0.0/16
 34.0.240.0/20
 34.104.116.0/22
 34.116.128.0/17
 34.118.0.0/17
 34.124.52.0/22
 34.88.0.0/16
 34.104.96.0/21
 34.124.32.0/21
 35.203.232.0/21
 35.217.0.0/18
 35.220.26.0/24
 35.228.0.0/16
 35.242.26.0/24
 34.0.192.0/19
 34.157.44.0/23
 34.157.172.0/23
 34.164.0.0/16
 34.175.0.0/16
 8.34.208.0/23
 8.34.211.0/24
 8.34.220.0/22
 23.251.128.0/20
 34.22.112.0/20
 34.22.128.0/17
 34.34.128.0/18
 34.38.0.0/16
 34.76.0.0/14
 34.118.254.0/23
 34.140.0.0/16
 35.187.0.0/17
 35.187.160.0/19
 35.189.192.0/18
 35.190.192.0/19
 35.195.0.0/16
 35.205.0.0/16
 35.206.128.0/18
 35.210.0.0/16
 35.220.96.0/19
 35.233.0.0/17
 35.240.0.0/17
 35.241.128.0/17
 35.242.64.0/19
 104.155.0.0/17
 104.199.0.0/18
 104.199.66.0/23
 104.199.68.0/22
 104.199.72.0/21
 104.199.80.0/20
 104.199.96.0/20
 130.211.48.0/20
 130.211.64.0/19
 130.211.96.0/20
 146.148.2.0/23
 146.148.4.0/22
 146.148.8.0/21
 146.148.16.0/20
 146.148.112.0/20
 192.158.28.0/22
 34.1.160.0/20
 34.32.0.0/17
 34.152.80.0/23
 34.177.36.0/23
 34.1.144.0/20
 34.17.0.0/16
 34.157.124.0/23
 34.157.250.0/23
 34.39.0.0/17
 34.89.0.0/17
 34.105.128.0/17
 34.127.186.0/23
 34.128.52.0/22
 34.142.0.0/17
 34.147.128.0/17
 34.157.36.0/22
 34.157.40.0/22
 34.157.168.0/22
 35.189.64.0/18
 35.197.192.0/18
 35.203.210.0/23
 35.203.212.0/22
 35.203.216.0/22
 35.214.0.0/17
 35.220.20.0/22
 35.230.128.0/19
 35.234.128.0/19
 35.235.48.0/20
 35.242.20.0/22
 35.242.128.0/18
 35.246.0.0/17
 34.0.224.0/24
 34.0.226.0/24
 34.40.0.0/17
 34.89.128.0/17
 34.104.112.0/23
 34.107.0.0/17
 34.118.244.0/22
 34.124.48.0/23
 34.141.0.0/17
 34.157.48.0/20
 34.157.176.0/20
 34.159.0.0/16
 35.198.64.0/18
 35.198.128.0/18
 35.207.64.0/18
 35.207.128.0/18
 35.220.18.0/23
 35.234.64.0/18
 35.235.32.0/20
 35.242.18.0/23
 35.242.192.0/18
 35.246.128.0/17
 34.1.224.0/19
 34.32.128.0/17
 34.34.0.0/17
 34.90.0.0/15
 34.104.126.0/23
 34.124.62.0/23
 34.141.128.0/17
 34.147.0.0/17
 34.157.80.0/23
 34.157.92.0/22
 34.157.208.0/23
 34.157.220.0/22
 35.204.0.0/16
 35.214.128.0/17
 35.220.16.0/23
 35.234.160.0/20
 35.242.16.0/23
 34.65.0.0/16
 34.104.110.0/23
 34.124.46.0/23
 35.216.128.0/17
 35.220.44.0/24
 35.235.216.0/21
 35.242.44.0/24
 34.0.160.0/19
 34.153.38.0/24
 34.153.230.0/24
 34.154.0.0/16
 34.157.8.0/23
 34.157.121.0/24
 34.157.136.0/23
 34.157.249.0/24
 35.219.224.0/19
 34.1.0.0/20
 34.155.0.0/16
 34.157.12.0/22
 34.157.140.0/22
 34.163.0.0/16
 34.36.0.0/16
 34.49.0.0/16
 34.54.0.0/16
 34.95.64.0/18
 34.96.64.0/18
 34.98.64.0/18
 34.102.128.0/17
 34.104.27.0/24
 34.107.128.0/17
 34.110.128.0/17
 34.111.0.0/16
 34.116.0.0/21
 34.117.0.0/16
 34.120.0.0/16
 34.128.128.0/18
 34.144.192.0/18
 34.149.0.0/16
 34.160.0.0/16
 35.186.192.0/18
 35.190.0.0/18
 35.190.64.0/19
 35.190.112.0/20
 35.201.64.0/18
 35.227.192.0/18
 35.241.0.0/18
 35.244.128.0/17
 107.178.240.0/20
 130.211.4.0/22
 130.211.8.0/21
 130.211.16.0/20
 130.211.32.0/20
 34.1.32.0/20
 34.18.0.0/16
 34.157.126.0/23
 34.157.252.0/23
 34.1.48.0/20
 34.152.84.0/23
 34.166.0.0/16
 34.177.48.0/23
 34.0.64.0/19
 34.157.90.0/23
 34.157.216.0/23
 34.165.0.0/16
 34.19.128.0/17
 34.20.0.0/17
 34.47.0.0/18
 34.95.0.0/18
 34.104.76.0/22
 34.118.128.0/18
 34.124.12.0/22
 34.128.37.0/24
 34.128.42.0/23
 34.128.49.0/24
 34.128.58.0/23
 34.152.0.0/18
 35.203.0.0/17
 35.215.0.0/18
 35.220.43.0/24
 35.234.240.0/20
 35.242.43.0/24
 34.0.32.0/20
 34.104.114.0/23
 34.124.50.0/23
 34.124.112.0/20
 34.130.0.0/16
 34.152.69.0/24
 34.157.255.0/24
 34.51.0.0/17
 34.153.42.0/23
 34.153.234.0/23
 34.39.128.0/17
 34.95.128.0/17
 34.104.80.0/21
 34.124.16.0/21
 34.151.0.0/18
 34.151.192.0/18
 35.198.0.0/18
 35.199.64.0/18
 35.215.192.0/18
 35.220.40.0/24
 35.235.0.0/20
 35.242.40.0/24
 35.247.192.0/18
 34.0.48.0/20
 34.104.50.0/23
 34.127.178.0/23
 34.153.33.0/24
 34.153.225.0/24
 34.176.0.0/16
 8.34.210.0/24
 8.34.212.0/22
 8.34.216.0/22
 8.35.192.0/21
 23.236.48.0/20
 23.251.144.0/20
 34.0.225.0/24
 34.16.0.0/17
 34.27.0.0/16
 34.28.0.0/14
 34.33.0.0/16
 34.41.0.0/16
 34.42.0.0/16
 34.44.0.0/15
 34.46.0.0/16
 34.66.0.0/15
 34.68.0.0/14
 34.72.0.0/16
 34.118.200.0/21
 34.121.0.0/16
 34.122.0.0/15
 34.128.32.0/22
 34.132.0.0/14
 34.136.0.0/16
 34.153.48.0/21
 34.153.240.0/21
 34.157.84.0/23
 34.157.96.0/20
 34.157.212.0/23
 34.157.224.0/20
 34.170.0.0/15
 34.172.0.0/15
 34.177.52.0/22
 35.184.0.0/16
 35.188.0.0/17
 35.188.128.0/18
 35.188.192.0/19
 35.192.0.0/15
 35.194.0.0/18
 35.202.0.0/16
 35.206.64.0/18
 35.208.0.0/15
 35.220.64.0/19
 35.222.0.0/15
 35.224.0.0/15
 35.226.0.0/16
 35.232.0.0/16
 35.238.0.0/15
 35.242.96.0/19
 104.154.16.0/20
 104.154.32.0/19
 104.154.64.0/19
 104.154.96.0/20
 104.154.113.0/24
 104.154.114.0/23
 104.154.116.0/22
 104.154.120.0/23
 104.154.128.0/17
 104.155.128.0/18
 104.197.0.0/16
 104.198.16.0/20
 104.198.32.0/19
 104.198.64.0/20
 104.198.128.0/17
 107.178.208.0/20
 108.59.80.0/21
 130.211.112.0/20
 130.211.128.0/18
 130.211.192.0/19
 130.211.224.0/20
 146.148.32.0/19
 146.148.64.0/19
 146.148.96.0/20
 162.222.176.0/21
 173.255.112.0/21
 199.192.115.0/24
 199.223.232.0/22
 199.223.236.0/24
 34.22.0.0/19
 35.186.0.0/17
 35.186.128.0/20
 35.206.32.0/19
 35.220.46.0/24
 35.242.46.0/24
 107.167.160.0/20
 108.59.88.0/21
 173.255.120.0/21
 34.23.0.0/16
 34.24.0.0/15
 34.26.0.0/16
 34.73.0.0/16
 34.74.0.0/15
 34.98.128.0/21
 34.118.250.0/23
 34.138.0.0/15
 34.148.0.0/16
 34.152.72.0/21
 34.177.40.0/21
 35.185.0.0/17
 35.190.128.0/18
 35.196.0.0/16
 35.207.0.0/18
 35.211.0.0/16
 35.220.0.0/20
 35.227.0.0/17
 35.229.16.0/20
 35.229.32.0/19
 35.229.64.0/18
 35.231.0.0/16
 35.237.0.0/16
 35.242.0.0/20
 35.243.128.0/17
 104.196.0.0/18
 104.196.65.0/24
 104.196.66.0/23
 104.196.68.0/22
 104.196.96.0/19
 104.196.128.0/18
 104.196.192.0/19
 162.216.148.0/22
 34.21.0.0/17
 34.48.0.0/16
 34.85.128.0/17
 34.86.0.0/16
 34.104.60.0/23
 34.104.124.0/23
 34.118.252.0/23
 34.124.60.0/23
 34.127.188.0/23
 34.145.128.0/17
 34.150.128.0/17
 34.157.0.0/21
 34.157.16.0/20
 34.157.128.0/21
 34.157.144.0/20
 35.186.160.0/19
 35.188.224.0/19
 35.194.64.0/19
 35.199.0.0/18
 35.212.0.0/17
 35.220.60.0/22
 35.221.0.0/18
 35.230.160.0/19
 35.234.176.0/20
 35.236.192.0/18
 35.242.60.0/22
 35.243.40.0/21
 35.245.0.0/16
 34.1.16.0/20
 34.157.32.0/22
 34.157.160.0/22
 34.162.0.0/16
 34.104.56.0/23
 34.127.184.0/23
 34.161.0.0/16
 35.206.10.0/23
 34.0.128.0/19
 34.157.46.0/23
 34.157.174.0/23
 34.174.0.0/16
 34.19.0.0/17
 34.53.0.0/17
 34.82.0.0/15
 34.105.0.0/17
 34.118.192.0/21
 34.127.0.0/17
 34.145.0.0/17
 34.157.112.0/21
 34.157.240.0/21
 34.168.0.0/15
 35.185.192.0/18
 35.197.0.0/17
 35.199.144.0/20
 35.199.160.0/19
 35.203.128.0/18
 35.212.128.0/17
 35.220.48.0/21
 35.227.128.0/18
 35.230.0.0/17
 35.233.128.0/17
 35.242.48.0/21
 35.243.32.0/21
 35.247.0.0/17
 104.196.224.0/19
 104.198.0.0/20
 104.198.96.0/20
 104.199.112.0/20
 34.20.128.0/17
 34.94.0.0/16
 34.102.0.0/17
 34.104.64.0/21
 34.108.0.0/16
 34.118.248.0/23
 34.124.0.0/21
 35.215.64.0/18
 35.220.47.0/24
 35.235.64.0/18
 35.236.0.0/17
 35.242.47.0/24
 35.243.0.0/21
 34.22.32.0/19
 34.104.52.0/24
 34.106.0.0/16
 34.127.180.0/24
 35.217.64.0/18
 35.220.31.0/24
 35.242.31.0/24
 34.16.128.0/17
 34.104.72.0/22
 34.118.240.0/22
 34.124.8.0/22
 34.125.0.0/16
 35.219.128.0/18
 34.37.0.0/16
 34.128.46.0/23
 34.128.62.0/23
 

Hetzner

5.9.0.0/16
5.75.128.0/17
5.161.0.0/21
5.161.164.0/22
5.161.168.0/21
5.161.238.0/23
5.222.0.0/15
23.88.0.0/17
37.27.0.0/16
45.145.227.0/24
46.4.0.0/16
46.62.128.0/17
49.12.0.0/15
65.21.0.0/16
65.108.0.0/15
77.42.0.0/17
78.46.0.0/15
78.138.62.0/24
85.10.192.0/18
88.99.0.0/16
88.198.0.0/16
89.167.0.0/17
91.99.0.0/16
91.107.128.0/17
91.190.240.0/21
91.233.8.0/22
94.130.0.0/16
95.216.0.0/15
116.202.0.0/15
128.140.0.0/17
135.181.0.0/16
136.243.0.0/16
138.199.128.0/17
138.201.0.0/16
142.132.128.0/17
144.76.0.0/16
148.251.0.0/16
157.90.0.0/16
157.180.0.0/17
159.69.0.0/16
162.55.0.0/16
167.233.0.0/16
167.235.0.0/16
168.119.0.0/16
171.25.225.0/24
176.9.0.0/16
178.63.0.0/16
178.212.75.0/24
185.12.64.0/22
185.50.120.0/23
185.107.52.0/22
185.126.28.0/22
185.157.83.0/24
185.157.176.0/22
185.171.224.0/22
185.189.228.0/22
185.213.45.0/24
185.216.237.0/24
185.226.99.0/24
185.228.8.0/23
185.242.76.0/24
185.253.111.0/24
188.34.128.0/17
188.40.0.0/16
188.245.0.0/16
193.25.170.0/23
193.110.6.0/23
193.163.198.0/24
194.42.180.0/22
194.42.184.0/22
194.62.106.0/24
195.60.226.0/24
195.201.0.0/16
195.248.224.0/24
197.242.84.0/22
201.131.3.0/24
204.29.146.0/24
213.133.96.0/19
213.232.193.0/24
213.239.192.0/18
216.55.108.0/22

ahrefs (SEO tool)

You can often see your logs dominated by their IPs.  Block all of ahrefs listed IPs.

On any random day, a lot of sites will receive quite a bit of traffic and high frequency queries from ahrefs.  If you aren't using their SEO service there is no point in allowing their traffic.

54.36.148.0/24
54.36.149.0/24
 
195.154.122.0/24
195.154.123.0/24
195.154.126.0/24
195.154.127.0/24
 
51.222.253.0/26

167.114.139.0/24
54.39.6.0/24
54.39.136.0/24
142.44.233.0/24
54.39.0.0/24
51.161.37.0/24
142.44.220.0/24
54.39.210.0/24
54.39.203.0/24
148.113.130.0/24
51.222.95.0/24
51.161.65.0/24
54.39.89.0/24
15.235.27.0/24
148.113.128.0/24
15.235.96.0/24
142.44.228.0/24
142.44.225.0/24
51.222.168.0/24
15.235.98.0/24
 
54.37.118.64/27
51.75.236.128/27
92.222.108.96/27
51.68.247.192/27
176.31.139.0/27
5.39.1.224/27
92.222.104.192/27
37.59.204.128/27
94.23.188.192/27
5.39.109.160/27
 
51.195.244.0/24
198.244.168.0/24
198.244.226.0/24
51.195.183.0/24
198.244.183.0/24
54.38.147.0/24
198.244.240.0/24
198.244.242.0/24
51.89.129.0/24
51.195.215.0/24

168.119.68.0/24
198.244.186.0/24
202.94.84.0/24

168.119.64.0/24
168.119.65.0/24

Uses:

• Install OS on physical drive
• Linux installation VMware
• Install Windows on physical drive
• Boot OS from physical drive
• VMware Workstation tutorial
• Linux virtual machine to physical disk
• Windows to Linux migration
• Dual boot OS installation
• Boot from external SSD
• Linux, Windows, BSD installation guide
• Physical drive boot OS
• Create bootable OS drive VMware
• Install OS on external drive

Description:

"Learn how to install any operating system—Linux, BSD, Windows, and more—on a physical drive from within Windows using VMware Workstation! This step-by-step guide shows you how to set up any OS on an external or internal drive and boot directly from it. Whether you're looking to dual boot or completely migrate from Windows to a new OS, this tutorial covers everything you need. You don’t need a bootable USB drive—everything is done directly through VMware Workstation on Windows.

🔹 What you’ll learn:

• Installing any OS (Linux, BSD, Windows, etc.) on a physical disk using VMware Workstation
• Creating a bootable drive from Windows
• Dual boot setup with Windows and another OS
• Setting up any OS to boot from an external or internal drive
• Troubleshooting common installation issues

🔹 Tools needed:

• VMware Workstation
• OS ISO file (Linux, BSD, Windows, etc.)
• Physical drive (internal/external HDD or SSD)

Youtube Video of Installing Linux to a physical drive from VMWare Workstation in Windows.

Step 1 - Open the "Disk Management Tool" in Windows

In this example there is a disk 0 and disk 1.  The easiest way is to verify by size, however if both drives are similar in size then the better way is to right click on each disk and click "Properties"

As we can see below Disk 0 is a WDC drive ( Western Digital ), if we know there's no other Western Digital drive and this is the one we want to install our OS to, choose this drive.

Step 2 - Create a new VM in VMWare Workstation

Open VMWare Workstation as "Administrator" and click "File -> New Virtual Machine" to create a new VM.

Step 3 - Create VM as "Custom (advanced)"

This just has the advantage of allowing us to have the Physical Drive attached initially as the only drive in VMWare rather than going back and modifying the storage settings/HDDs later on.

 Step 4 - Choose the OS

In our example we are installing Ubuntu 64 bit, it's important to choose the correct OS type and even more important to choose the right architecture (eg 64 bit).  This becomes more apparent later on if the intended computer you want to boot your installed OS on uses EFI (EFI mode can only be enabled easily when there is the 64-bit architecture selected).

 Step 5 - Choose Your Physical Disk

This part is crucial and critical, if you mess this up you could delete your crypto keys or family photos by accident!  If possible make sure all other drives/storage devices are disconnected or disabled by this time.

• Click "Use physical disk (for advanced users)" and then we'll select the physical drive#
• Remember that the drive number corresponds to the drive we identify in Disk Management (in our case we want to install to Drive 0 which we showed earlier is our WDC drive).

Make sure that you make proper note of the physical drive so you don't wipe the wrong drive and lose data!

Once again we choose PhysicalDrive0 because it corresponds to the identified "Disk 0" in Disk Management.  You need to verify your physical drive number or you may wipe your data on an unintended drive.

After this point all the defaults are normally fine (continue to click next through the menu to create the VM).

Step 6 - Set EFI boot mode if you need

If your target device will be booting by EFI you need to edit your VM by going to Settings and then "Options" and "Advanced" and choose UEFI as the boot mode.  This is not necessary if you will be booting by MBR (however many newer systems will only boot EFI)

Step 7 - Boot your .iso and install to your physical drive.

Install your OS like it's a normal physical machine, once the install is completed, power off the VM and you can now boot the drive you installed to on the same or other computer.

Your frontend CDN (eg. Cloudflare or even your own load balancer/proxy) must be sending the X-Forwarded-For and you must be running Apache on the backend.
This solves the problem where your logs and services will only see the proxy/CDN IP and not the real client IP.

mod_rpaf will fix all of this

This solution transparently sets the real IP of the client for Apache and any services that rely on the REMOTE_ADDR without having to modify any code.
 

Install the right tools:


#install the devel tools and gcc for your distro and git
yum -y install httpd-devel gcc git

 

Get the mod_rpaf source code

#let's git the mod_rpaf module (most distros don't have this as a package installable module) so we have to build it ourselves
cd ~
git clone https://github.com/gnif/mod_rpaf.git
cd mod_rpaf

 

Compile mod_rpaf


#let's compile and install the mod_rpaf.c code
apxs -i -c -n rpaf mod_rpaf.c

 

Edit httpd.conf

#edit the httpd.conf for Apache to enable mod_rpaf, be sure to set the RPAF_ProxyIPs to whatever IPs your CDN/proxy uses otherwise the original CDN IP will still be presented to Apache

Edit the two example of IPs below to be your CDN/Proxy/LoadBalancer IPs (if you only have 1 IP then only put 1 there).

LoadModule rpaf_module modules/mod_rpaf.so


    RPAF_Enable             On
    RPAF_Header             X-Forwarded-For
    RPAF_ProxyIPs           1.2.3.4 5.6.7.8
    RPAF_SetHostName        On
    RPAF_SetHTTPS           On
    RPAF_SetPort            On


 

#remember to restart httpd/apache


systemctl restart httpd
#now remove gcc and httpd-devel and git for security reasons
yum -y remove gcc httpd-devel git

 yum update
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
Could not retrieve mirrorlist http://mirrorlist.centos.org/?release=7&arch=x86_64&repo=os&infra=container error was
14: curl#6 - "Could not resolve host: mirrorlist.centos.org; Unknown error"


 One of the configured repositories failed (Unknown),
 and yum doesn't have enough cached data to continue. At this point the only
 safe thing yum can do is fail. There are a few ways to work "fix" this:

     1. Contact the upstream for the repository and get them to fix the problem.

     2. Reconfigure the baseurl/etc. for the repository, to point to a working
        upstream. This is most often useful if you are using a newer
        distribution release than is supported by the repository (and the
        packages for the previous distribution release still work).

     3. Run the command with the repository temporarily disabled
            yum --disablerepo= ...

     4. Disable the repository permanently, so yum won't use it by default. Yum
        will then just ignore the repository until you permanently enable it
        again or use --enablerepo for temporary usage:

            yum-config-manager --disable
        or
            subscription-manager repos --disable=

     5. Configure the failing repository to be skipped, if it is unavailable.
        Note that yum will try to contact the repo. when it runs most commands,
        so will have to try and fail each time (and thus. yum will be be much
        slower). If it is a very temporary problem though, this is often a nice
        compromise:

            yum-config-manager --save --setopt=.skip_if_unavailable=true

Cannot find a valid baseurl for repo: base/7/x86_64
 

The issue is that CentOS 7 went EOL in June 2024, but you can still access the packages in the CentOS vault for the last 7.9.2009 release for testing purposes or to create an environment to update to AlmaLinux or another newer distro.


Solution - Replace the contents of the Base Repo file "/etc/yum.repos.d/CentOS-Base.repo" as below:




[base]
name=CentOS-$releasever - Base
baseurl=http://vault.centos.org/7.9.2009/os/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[updates]
name=CentOS-$releasever - Updates
baseurl=http://vault.centos.org/7.9.2009/updates/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[extras]
name=CentOS-$releasever - Extras
baseurl=http://vault.centos.org/7.9.2009/extras/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[centosplus]
name=CentOS-$releasever - Plus
baseurl=http://vault.centos.org/7.9.2009/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7


Solution for the SCLo repo

yum install centos-release-scl

/etc/yum.repos.d/CentOS-SCLo-scl.repo

[centos-sclo-sclo]
name=CentOS-7 - SCLo sclo
baseurl=http://vault.centos.org/7.9.2009/sclo/$basearch/sclo/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[centos-sclo-sclo-source]
name=CentOS-7 - SCLo sclo Source
baseurl=http://vault.centos.org/7.9.2009/sclo/Source/sclo/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
 

/etc/yum.repos.d/CentOS-SCLo-scl      

[centos-sclo-rh]
name=CentOS-7 - SCLo rh
baseurl=http://vault.centos.org/7.9.2009/sclo/$basearch/rh/
gpgcheck=1
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7

[centos-sclo-rh-source]
name=CentOS-7 - SCLo rh Source
baseurl=http://vault.centos.org/7.9.2009/sclo/Source/rh/
gpgcheck=1
enabled=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-7
 

A lot of times you may see when installing packages that apt will recommend packages, a lot of times when doing things like upgrading or install a new kernel, it is very smart at recommending related packages (eg. modules, headers, extras etc..)

All you need to do is add this flag to your "apt install" command:

--install-suggests

As you can see below, when we add this flag, it then installs all of the "Suggested packages" from above:

sudo apt install --install-suggests linux-image-5.15.0-119-generic
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  linux-objects-nvidia-510-4.15.0-188-generic linux-signatures-nvidia-4.15.0-188-generic
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  fdutils linux-doc linux-headers-5.15.0-119-generic linux-hwe-5.15-headers-5.15.0-119 linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic
Suggested packages:
  linux-hwe-5.15-tools
The following NEW packages will be installed:
  fdutils linux-doc linux-headers-5.15.0-119-generic linux-hwe-5.15-headers-5.15.0-119 linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic


Just export these variables:


export http_proxy="socks5://127.0.0.1:1234"
export https_proxy="socks5://127.0.0.1:1234"

The above example uses a socks 5 proxy at localhost and port 1234.

This is a great way to improve performance of some activities for things like package installation, especially if your routing is not ideal to the repos.

Do you have a PDF that was too large after merging using convert / ImageMagick?

Use gs or ghostscript as below, specify the output file and also the input file.

In the example below the output file is "outputfile-resized.pdf" and the input file is "original-pdf.pdf".  Change the input file to the name of your pdf

gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/prepress -dNOPAUSE -dQUIET -dBATCH -sOutputFile=outputfile-resized.pdf original-pdf.pdf

For example the above method took a pdf from 12MB to about 500KB with no noticeable quality loss.

qemu: could not load PC BIOS 'bios-256k.bin'


The file exists here:

/usr/share/qemu/bios-256k.bin

It is symlinked to here:

/usr/share/seabios/bios.bin
 

A quick fix is manually specifying the BIOS:

 -bios /usr/share/qemu/bios-256k.bin

If you just have 1 disk per node/testing/POC then you can reduce the overall disk usage during install as below:

1.) Click on "Options" on the HDD that you want to install to.

2.) Under "hdsize" Set the Maximum Size smaller than the disk size.

In this example the disk is 60G and I set max size as 20G.  This then leaves 40G remaining of unpartitioned space that you could later use for other things like Ceph, DRBD etc...

Hyper-V uses obscure virtual hardware, you can enable the FB device for Hyper-V but if you just need text/console to work just compile this option:

Device Drivers --->
  Graphics support --->
    <*> Framebuffer Devices --->
        <*> Simple framebuffer support



If you have ever seen a system where terminal in the GUI closes instantly and/or you cannot SSH to the server/machine.

mount -t devtmpfs none /dev
mount -t devpts none /dev/pts

As a fun test in an old/unloved dual CPU Xeon system, I removed the fan for the heatsink and after 1.5 hours, the system was completely stable and had surprisingly not crashed, despite exceeding or hitting the critical thermal levels

If you are sure the binary/program exists, it could be that your PATH variable is broken or not set.

Just do this to set most of the common paths:

export PATH=$PATH:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

There's no risk to the above, because it just takes your current PATH variable and adds more to what is already existing.

To make this permanent you can add this to your ~/.bashrc file

In a lot of distros, vdpau which accelerates video through the GPU is often not installed even in recent distros.  This will result in Firefox with Youtube dropping frames when you check the "Stats for Nerds" and to the eye as jerky or slower video playback due to the dropped frames.

Some sites will say that because you are watching VP09 that there is no GPU support, but with vdpau installed and enabled, many GPUs will still be able to use hardware decoding for 4K Youtube videos.  Information on this is spotty but you can clearly see the GPU utilization going up when watching Youtube GPUs, so the GPU is doing something to decode the video.

This will make you think that your GPU is not capable of playing even 1080P sometimes or 4K even though it is, as in many cases it is your CPU doing the video decoding and a lot of times some CPUs cannot reliably or consistently support 4K decoding in Youtube.

The easiest solution is to install vdpau and restart your browser:

sudo apt install vdpau-driver-all

I was surprised that this helped even on even much older cards including older Nvidia GPUs.

After doing the above and restarting Firefox, the stats for nerds showed 0 dropped frames when watching 4K, compared to some dropped frames at 1080P and many dropped at 4K before enabling vdpau.

One caveat that it may be that your video card is still too slow and may not be able to support decoding Youtube in fullscreen at certain resolutions.  You may see the above but then on a weaker GPU, may find that it does drop frames when running fullscreen.

Sometimes frames will only drop when having the Stats for nerds on top, or the progress bar open.

In general VP09 full support and decoding is not guaranteed to work well or at all, and unfortunately Youtube has not stuck to normal standards that most GPUs support like H265 or H264, XVID, DIVX etc...

Related Links:

https://support.mozilla.org/en-US/kb/firefox-4k-ultra-high-definition-youtube-videos

https://www.nvidia.com/en-us/geforce/forums/geforce-graphics-cards/5/285268/no-hardware-decoding-on-youtube-with-vp9-hdr/

https://www.reddit.com/r/linuxquestions/comments/ugp19e/help_playing_back_4k60fps_youtube_on_linux/

This example is using RTL8821AU chipset from a TP-Link T2U Plus USB adapter and other similar ones should work the same.

Here's how to get this chipset RTL8821AU working in Linux.

Here's how to get RTL8812 and RTL8822 working in Linux.

With these settings I can get very good speeds of 300+ mbit per second and for some reason the upload speed is faster than download.

Change wlan0 if your adapter has a different device name and make sure you set your WPA passphrase or change it.

interface=wlan0
# "a" simply means 5GHz
#driver=cfg80211
hw_mode=a
# the channel to use, 0 means the AP will search for the channel with the least interferences (ACS)
channel=36
#ieee80211d=1
country_code=US
ieee80211n=1
ht_capab=[HT40+][HT40-][SHORT-GI-20][SHORT-GI-40][MAX-AMSDU-7935]
#the below doesn't work hostapd wouldn't start
#ht_capab=[HT40+][HT40-][SHORT-GI-20][SHORT-GI-40][RX-STBC1][MAX-AMSDU-7935]
# 802.11ac support
ieee80211ac=1         
wmm_enabled=1
#vht_capab=[MAX-MPDU-11454][SHORT-GI-80][HTC-VHT]
vht_capab=[MAX-MPDU-11454][SHORT-GI-80][RX-STBC-1][HTC-VHT][MAX-A-MPDU-LEN-EXP7]

#optimize?
beacon_int=100
dtim_period=2
max_num_sta=32
macaddr_acl=0
rts_threshold=2347
fragm_threshold=2346

#end optimize
ssid=rttWirelessInternet
auth_algs=1
wpa=2
wpa_key_mgmt=WPA-PSK
rsn_pairwise=CCMP
wpa_passphrase=YOURWirelessInSecurePasswordHere


Edit your /etc/hostapd/hostapd.conf file like below and restart hostapd.

#WPS stuff
# Enable control interface for PBC/PIN entry


wpa_psk_file=/etc/hostapd/hostapd.psk
ctrl_interface=/var/run/hostapd
eap_server=1
wps_state=2
ap_setup_locked=1
wps_pin_requests=/var/run/hostapd.pin-req
config_methods=label display push_button keypad

#WPS model info stuff change to suit your needs
device_name=Your Device Name
manufacturer=Company
model_name=Model
model_number=ModelNum

Create the .psk file from above

touch /etc/hostapd/hostapd.psk

Restart hostapd

systemctl restart hostapd
 

Here's how to push the non-existent button on your router for WPS

hostapd_cli wps_pbc

Then you should get this and your device should be able to connect now.

Once you see the below, the device has 2 minutes to connect.

Selected interface 'wlan0'
OK


This is certainly a poor design, as you can read many seasoned admins who have updated their iDRAC only to have it killed.  One possible cause is not by doing all the incremental updates, doing updates from an old iDRAC to one many revisions newer is a sure way to kill things, but even then there is no guarantee based on the amount of failures.

List of threads of people's dead iDRAC's:

https://www.reddit.com/r/homelab/comments/a49b4y/r720_idrac_help/?rdt=40792

https://www.reddit.com/r/homelab/comments/960wx8/dell_r720_idrac_recovery/

https://www.dell.com/community/en/conversations/rack-servers/idrac-7-not-responding-on-dell-poweredge-r720xd/

https://www.dell.com/community/en/conversations/poweredge-hardware-general/iidrac-initialization-error-r720/647f86a0f4ccf8a8de5b28ca

https://www.dell.com/community/en/conversations/poweredge-hardware-general/alert-idrac-not-responding-rebooting/

https://community.spiceworks.com/t/alert-idrac-not-responding-rebooting/776991

https://www.dell.com/community/en/conversations/systems-management-general/dead-idrac-after-update/647f932df4ccf8a8de53107b

The first solution is don't mass update, as we've seen entire datacenters or racks full of Dell machines with dead iDRAC due to mass update attempts.  It is likely not practical to do this fix on a wide scale unless you can migrate the applications to another facility or rack as this requires the servers to be pulled off the rack or at least pulled by the rails and to open the case of course.

Try This First (only if you have lights on the iDRAC):

https://www.dell.com/support/kbdoc/en-us/000120131/poweredge-idrac-recovery-procedure-with-firmimg-d7

This solution is for if your iDRAC is totally dead, no flashing lights, dead NIC etc...

You will first need a USB to TTL adapter like this Amazon affil link.

Story of successful recovery.

Another successful recovery: http://blog.ignoranthack.me/?p=86

More discussion of the issue.

You will see Dell themselves advising to replace the motherboard, but in many cases by connecting to the UART below with a USB to TTL cable, you can tell the iDRAC to reflash from the SDCARD image.

Based off this example of a Dell R720, we can see the UART pins which are labelled by this github user.

This area is at the back right of the server (if looking from the server from the front) and will require you to remove the riser cage to access this area.

Step 1.) Connect to USB to TTL adapter.

Be sure to use the correct pins below (remember the TX side of the TTL goes to the RX side of the UART and the the RX side of the TTL goes to the TX side of the UART).  If in doubt, use a multimeter to make sure you aren't connecting to the wrong pins/getting voltage where you shouldn't be.

Note About Format of SD Card

I have seen this occur on some servers, I am not sure how it was done like this but the SD Card came formatted as the root drive, without any partition.  It seems that U-Boot recovery DOES NOT like this, or the fact that it is a 16GB Card even though it is from Dell.


UTIL RECOVER:Transport:sd  TargetMMC:EMMC  File:firmimg.d7
UTIL RECOVER:ERROR:Could not locate 'firmimg.d7' in FAT or EXT2 file system.


If you get the errors above and you are sure the file exists, then it is worth doing the following.

1.) Backup all files from the SD Card.

2.) Completely wipe the card.

3.) Format a single partition as 2GB as ext2 (ext2 not ext3 or ext4 or it won't work as the environment does not have the kernel modules to load anything but FAT and EXT2).

Step 2.) Recover

Sometimes you may need to go through 2 rounds of bootup procedure by hitting all the annoying keys to get the server booted normally for this to work.

Connect using minicom or screen and run these commands:

screen /dev/ttyUSB0 115200

The connection speed is crucial, otherwise you will see nothing or garbage characters.

If all goes well you should see like below.

Make sure you have the file firmimg.d7 or whatever the name one is from the iDRAC firmware on the SD card that inserts at the front of the server.

util recover -emmc -from sd -f firmimg.d7 -noreset -clear

Some versions will not have this -from_sd option

iDRAC8=> util recover -emmc -from_sd -f firmimg.d6 -noreset -clear
util recover:mcgetopt:ERROR: option "-from_sd" was not found.

Then once the above completes do this:

util reset

Short iDRAC DBG interrupt/reboot pins

If you see a shell prompt like below (SH7757), you will need to short the DBG pins in the image from earlier.

Use a paper clip to short the two pins shown above as the interrupt/reboot pins in the diagram above.

After shorting you should see U-Boot again:

U-Boot 2009.08 (Mar 10 2012 - 02:52:09) Avocent (0.0.3) EVB, Build: jenkins-12G_iDRAC_Releases-402

CPU: SH-4A  
BOARD: R0P7757LC00xxRL (C0 step) board
BOOT: Secure, HRK generated
DRAM:  240MB
  (240MB of 256MB total DRAM is available on U-Boot)
ENV:   Using primary env area.
In:    serial
Out:   serial
Err:   serial
PCIe:  Bridge loaded with 0x18000 bytes
WDT2:  Booted Lower Vector, 'uboot1'
sh_mmcif: 0, sh-sdhi: 1
Net:   sh_eth.0, sh_g_eth.0
INFO: 00:002 Start-up -to- util_idrac_main()
INFO: 00:004 U-Boot 2009.08 (Mar 10 2012 - 02:52:09) Avocent (0.0.3) EVB
INFO: 00:007 U-Boot checkin date(03-02-2012) Version(1.0.175)
INFO: 00:006 iDRAC PPID <NULL>
INFO: 00:003 SPI NOR init 4096 KiB N25Q032 bus=0 cs=0, speed=1000000, mode=3
INFO: 00:007 SH-4A Product: Major Ver=0x31  Minor Ver=0x13 C3 Little endian
             Family=0x10    Major Ver=0x30  Minor Ver=0x0b
PASS: 00:016 Dedicated monolithic mgmt NIC disabled
INFO: 00:128 BCM54610 OUI=0x3fffff  Model=0x3f  Revision=0x0f PhyAddr=1
INFO: 00:362 SD CARD:   Device: sh-sdhi   Manufacturer ID: 41   OEM: 3432
             Name: DELL1   Tran Speed: 25000000   Rd Block Len: 512
             SD version 2.0   High Capacity: Yes   Capacity: -843055104
INFO: 00:059 EMMC:   Device: sh_mmcif   Manufacturer ID: 90   OEM: 14a
             Name: HYNIX   Tran Speed: 25000000   Rd Block Len: 512
             MMC version 4.0   High Capacity: Yes   Capacity: 0
INFO: 00:019 CPLD: Major Ver=0x1  Minor Ver=0x0  Maint Ver=0x3
             Planar: Type=0x02  Rev=0x8  Rework=0x8  Scratch/PathRetry=0x00
PASS: 00:013 Coin cell detected good,  AD=0x397 low water=0x2c1
PASS: 00:008 PCIe C0 Ver=0.13 MCTP en, CRC=0x19592849 @0x8efbf3a8 cnt=0x18000
INFO: 00:007 Init PCIe mailbox(PCIe 0xFFEE0150=0x40010000)
INFO: 00:006 mode=normal
INFO: 00:002 reset_cause=ac
PASS: 00:005 Booted Lower Vector, 'uboot1' wdt2cnt=0
INFO: 00:005 wdt0cnt=0
PASS: 00:003 Clear CH1/CH2, clear 4K shared memory@0xffcaa000 on AC power up
PASS: 00:007 SMR0 no sermux env, default 0xd4
INFO: 00:004 GRACR=0x3c HISEL=0x00 SIRQCR5_D=0x03 SIRQCR6_D=0x01 LADMSK0=0xff2
             MRSTCR0=0xfedffe7f MRSTCR1=0xfff3ff0f MRSTCR2=0x7f80feff
             BARMAP=0x1 BCR=0x85000000 NCER=0x01fc NCMCR=0x0006 NCCSR=0x0303
PASS: 00:021 etherc0=78:45:C4:F6:88:68
             getherc0=78:45:C4:F6:88:69
INFO: 00:010 Fan logic for monolithic planar type 2
             fan1 - def 0000 1fff 3d   fan2 - def 0000 1fff 3d
             fan3 - def 0000 1fff 3d   fan4 - def 0000 1fff 3d
             fan5 - def 0000 1fff 3d   fan6 - def 0000 1fff 3d
             fan7 - def 0000 1fff 3d   fan8 - def 0000 1fff 3d
INFO: 00:076 Env and backup CRC'ed ok
*** no text signature found ***

INFO: 00:536 Sync eMMC/SPI NOR/Alternate u-boot images
PASS: 00:258 Current u-boot1 1.0.175 verified with 'ubootN'
             Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x0
             U-boot2 in sync with u-boot1 1.0.175
PASS: 03:607 Verify OS Images N: CRC32 OK: Kernel=0x4d21804a Rootfs=0x40007db7
PASS: 00:008 Boot device=emmc  Boot partition1/N
             Boot Path Retry:P1/N=0 P5/N-1=0
INFO: 00:000 05:198


WDT2: Disabled
Initialization complete, proceed with boot
*** no text signature found ***

No.  Device
---------------------
 1   EVB SD/MMC
 4   FLASH  

Current Device ==> 1

OS_WDT0_EN:180 seconds,  Re-load ILRAM:
Boot Path Retry:P1/N=0 P5/N-1=0
Copying kernel image ... OK
## Booting kernel from Legacy Image at 82000000 ...
   Image Name:   sh-linux
   Image Type:   SuperH Linux Kernel Image (gzip compressed)
   Data Size:    2535377 Bytes =  2.4 MB
   Load Address: 80001000
   Entry Point:  80002000
   Verifying Checksum ... OK
   Uncompressing Kernel Image ... OK
WDT2: Disable in do_bootm_linux()
sh_eth_init

Monolithic/DRB
init started: BusyBox v1.18.4 (2012-03-10 03:35:27 UTC)
starting pid 415, tty '': '/etc/init.d/rcS'
waitfor: pid 415 ...
setting net.ipv6.conf.default.accept_dad based on kernel bootup option
enable normal accept DAD for ipv6
Sat Jan 1 00:00:01 UTC 2000 To execute command __source /etc/sysapps_script/I_1000_mountfs.sh__
Sat Jan 1 00:00:03 UTC 2000 To execute command __source /etc/sysapps_script/I_1001_trigwdt.sh__
<========= Start H/W watchdog first... =========>
Sat Jan 1 00:00:03 UTC 2000 To execute command __source /etc/sysapps_script/I_1100_nfs.sh__
Sat Jan 1 00:00:03 UTC 2000 To execute command __source /etc/sysapps_script/I_1105_ps.sh__
XXXXXX /dev/mmcblk0p11
/dev/mmcblk0p11: recovering journal
fsck.ext3: Attempt to read block from filesystem resulted in short read while trying to re-open /dev/mmcblk0p11
e2fsck: io manager magic bad!
e2fsck 1.41.14 (22-Dec-2010)
fsck.ext3: Attempt to read block from filesystem resulted in short read while trying to open /dev/mmcblk0p11
Could this be a zero-length partition?
mount: mounting /dev/mmcblk0p11 on /flash/data0 failed: Invalid argument
mounting private storage fail, formatting partition
/etc/init.d/rcS: /etc/sysapps_script/I_1105_ps.sh: line 29: mkfs.ext3: Input/output error
/etc/init.d/rcS: /etc/sysapps_script/I_1105_ps.sh: line 29: tune2fs: Input/output error
mount: mounting /dev/mmcblk0p11 on /flash/data0 failed: Invalid argument
Sat Jan 1 00:00:04 UTC 2000 To execute command __source /etc/sysapps_script/I_1106_clear_ps.sh__
Sat Jan 1 00:00:04 UTC 2000 To execute command __source /etc/sysapps_script/I_1150_SYS_Drv.sh__
insmod: can't insert '/lib/modules/aess_eventhandlerdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_kcsdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_biospostdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_gpiodrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_sgpiodrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_dynairqdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_i2c_hwctrldrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_i2cdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_spi1drv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_pecisensordrv.ko': Input/output error
insmod: can't insert '/lib/modules/cryptodev.ko': Input/output error
insmod: read error from '/lib/modules/sh_tsip.ko': Input/output error
insmod: can't insert '/lib/modules/sh_tsip.ko': Input/output error
insmod: can't insert '/lib/modules/aess_fansensordrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_pwmdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_timerdrv.ko': Input/output error
insmod: can't insert '/lib/modules/dell_cplddrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_rspidrv.ko': Input/output error
insmod: can't insert '/lib/modules/dell_fpdrv.ko': Input/output error
insmod: can't insert '/lib/modules/aess_adcsensordrv.ko': Input/output error
insmod: can't insert '/lib/modules/VKCSDriver.ko': Input/output error
insmod: can't insert '/lib/modules/aess_memdrv.ko': Input/output error
insmod: read error from '/lib/modules/sh_pbi.ko': Input/output error
insmod: can't insert '/lib/modules/sh_pbi.ko': Input/output error
Sat Jan 1 00:00:04 UTC 2000 To execute command __source /etc/sysapps_script/I_1170_mknode.sh__
Sat Jan 1 00:00:05 UTC 2000 To execute command __/etc/sysapps_script/config_lib_init.sh__
mounting Platform Data 1 partition
mount: mounting /dev/mmcblk0p9 on /tmp/pd0/ failed: Input/output error
/etc/sysapps_script/config_lib_init.sh: line 28: /etc/default/ipmi/getsysid: not found
mkdir: can't create directory '/flash/data0/config/': Read-only file system
/etc/sysapps_script/config_lib_init.sh: cd: line 38: can't cd to /flash/data0/config/
ln: ./lmcfg.txt: Read-only file system
ln: ./platcfggrp.txt: Read-only file system
ln: ./platcfgfld.txt: Read-only file system
ln: ./cfgfld.txt: Read-only file system
ln: ./cfggrp.txt: Read-only file system
ln: ./gencfggrp.txt: Read-only file system
ln: ./gencfgfld.txt: Read-only file system
ln: ./altdefaults.txt: Read-only file system
cp: can't stat '/tmp/pd0/network_config/Orca/iDRACnet.default': No such file or directory
cp: can't stat '/tmp/pd0/network_config/Orca/NICSelection.sh': No such file or directory

Sat Jan 1 00:00:06 UTC 2000 To execute command __source /etc/sysapps_script/I_1160_OSINET.sh__
Sat Jan 1 00:00:06 UTC 2000 To execute command __source /etc/sysapps_script/I_1200_etc_defaults.sh__
cp: can't create '/flash/data0/etc/hosts': Read-only file system
mkdir: can't create directory '/flash/data0/sysconfig': Read-only file system
mkdir: can't create directory '/flash/data0/ipsec': Read-only file system
/etc/sysapps_script/network_dev_init.sh: line 34: /tmp/pd0/ipmi/getsysid: not found
cp: can't create '/flash/data0/etc/hosts': Read-only file system
Sat Jan 1 00:00:06 UTC 2000 To execute command __source /etc/sysapps_script/I_1210_sysklog_syslogd.sh__
mkdir: can't create directory '/var/log/raclogd/': No such file or directory
touch: /var/log/raclogd/raclog: No such file or directory
chmod: /var/log/raclogd/raclog: No such file or directory
/etc/init.d/rcS: /etc/sysapps_script/I_1210_sysklog_syslogd.sh: line 16: /etc/sysapps_script/syslog.sh: Input/output error
Sat Jan 1 00:00:07 UTC 2000 To execute command __source /etc/sysapps_script/I_trab_asdf.sh__
insmod: read error from '/lib/modules/trab.ko': Input/output error
insmod: can't insert '/lib/modules/trab.ko': Input/output error
Sat Jan 1 00:00:07 UTC 2000 To execute command __source /etc/sysapps_script/I_usb_drv.sh__
NOTICE: Not full features configuration - kbdmouse not operational
insmod: can't insert '/lib/modules/g_r8a66597_udc.ko': Input/output error
failed load of g_r8a66597_udc
Sat Jan 1 00:00:07 UTC 2000 To execute command __source /etc/sysapps_script/I_video_drv.sh__
insmod: can't insert '/lib/modules/aess_video.ko': Input/output error
waitfor: pid 415, wpid 415

Please press Enter to activate this console. ^[[5~^[[5~
starting pid 550, tty '': '-/bin/sh'




Other useful recon output of how things work

iDRAC8=> util recover


UTIL RECOVER:Transport:tftp  TargetMMC:EMMC  File:firmimg.d7
UTIL RECOVER:Loading file "firmimg.d7" from TFTP server IP "192.168.0.100"
reset PHY of sh_g_eth.0
sh_eth: phy reset timeout
sh_eth: phy config timeout
UTIL RECOVER:ERROR:TFTP load.
UTIL FAIL


iDRAC8 commands

iDRAC8=> util help
Available "util" subcommands:

evc  [-en ] [-dis ]
evc  [-rise] [-fall] [-monitor]

fan  [-set ]
fan  [-init]

flash  [-sd | -spi | -emmc]
       [-b |

iDRAC8=>
iDRAC8=> util recover -emmc -from sd -f firmimg.d6 -noreset -clear
util recover:mcgetopt:ERROR: option "-clear" was not found.

UTIL FAIL
iDRAC8=>
iDRAC8=> util recover -emmc -from sd -f firmimg.d6 -noreset       


UTIL RECOVER:Transport:sd  TargetMMC:EMMC  File:firmimg.d6
UTIL RECOVER:ERROR:Could not locate 'firmimg.d6' in FAT or EXT2 file system.

UTIL FAIL
iDRAC8=> util recover -emmc -from sd -f firmimg.d7 -noreset


UTIL RECOVER:Transport:sd  TargetMMC:EMMC  File:firmimg.d7
UTIL RECOVER:ERROR:Could not locate 'firmimg.d7' in FAT or EXT2 file system.


U-Boot 2009.08 (Jun 11 2018 - 05:56:29) Avocent (0.0.3) EVB

CPU: SH-4A
BOARD: R0P7757LC00xxRL (Cn step) board
BOOT: Secure, HRK generated
DRAM:  240MB
  (240MB of 256MB total DRAM is available on U-Boot)
ENV:   Using primary env area.
In:    serial
Out:   serial
Err:   serial
WDT2:  Booted Lower Vector, 'uboot1'
sh_mmcif: 0, sh-sdhi: 1
Net:   sh_eth.0, sh_g_eth.0
INFO: 00:002 Start-up -to- util_idrac_main()
INFO: 00:005 U-Boot 2009.08 (Jun 11 2018 - 05:56:29) Avocent (0.0.3) EVB
INFO: 00:006 U-Boot checkin date(08-11-2016) Version(1.1.32)
INFO: 00:006 iDRAC PPID
INFO: 00:003 SPI NOR init 4096 KiB N25Q032 bus=0 cs=0, speed=1000000, mode=3
INFO: 00:007 SH-4A Product: Major Ver=0x31  Minor Ver=0x13 C3 Little endian
             Family=0x10    Major Ver=0x30  Minor Ver=0x0b
INFO: 00:140 BCM54610 OUI=0x00d897  Model=0x26  Revision=0x0a PhyAddr=1
INFO: 00:006 Dedicated monolithic mgmt NIC disabled
INFO: 00:340 SD CARD:   Device: sh-sdhi   Manufacturer ID: 41   OEM: 3432
             Name: DELL1   Tran Speed: 25000000   Rd Blk Len: 512
             SD ver 2.0   HC: Yes   cap(<2gb): 3451912192  cap(>2gb): 0
             ext_rev: 0    rst_n: 0  ext[96]: 0
INFO: 00:063 EMMC:   Device: sh_mmcif   Manufacturer ID: 90   OEM: 14a
             Name: HYNIX   Tran Speed: 25000000   Rd Blk Len: 512
             MMC ver 4.0   HC: Yes   cap(<2gb): 0  cap(>2gb): 7569408
             ext_rev: 5    rst_n: 0  ext[96]: 0
INFO: 00:023 CPLD: Major Ver=0x1  Minor Ver=0x0  Maint Ver=0x3
             Planar: Type=0x02  Rev=0x8  Rework=0x8  Scratch/PathRetry=0x0f
PASS: 00:014 Coin cell detected good,  AD=0x397 low water=0x2c1
PASS: 00:008 PCIe SH7757_C0 Ver=0.15 MCTP en, CRC=0x8e9b6875 @0x8efbd914 cnt=0x18000
INFO: 00:008 Init PCIe mailbox(PCIe 0xFFEE0150=0x40010000)
INFO: 00:006 mode=normal
INFO: 00:002 reset_cause=recover
PASS: 00:005 Booted Lower Vector, 'uboot1' wdt2cnt=0
INFO: 00:006 wdt0cnt=0  cpld_recover_cnt=0  ddr_recover_cnt=0  
PASS: 00:006 SMR0 no sermux env, default 0xd4
INFO: 00:004 GRACR=0x3c HISEL=0x00 SIRQCR5_D=0x03 SIRQCR6_D=0x01 LADMSK0=0xff2
             MRSTCR0=0xfedffe7f MRSTCR1=0xfff3ff0f MRSTCR2=0x6f80feff
             BARMAP=0x1 BCR=0x85000000 NCER=0x01fc NCMCR=0x0006 NCCSR=0x0303
PASS: 00:021 etherc0=
             getherc0=
INFO: 00:008 Fan logic is not modified on Non-AC power up
INFO: 00:052 Env and backup CRC'ed ok
*** no text signature found ***
INFO: 00:696 Sync eMMC/SPI NOR/Alternate u-boot images
PASS: 00:077 Current u-boot1 1.1.32 verified with 'ubootN1'
             Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x0

FAIL: 06:813 Verify OS Images N-1: Rootfs crc exp=0xe8caa151 rec=0xcfb9c0a7
             blk_start=0x48003 blk_size=0x301f0 ENV bcnt=0x603e000
FAIL: 00:013 Boot device=emmc  Boot partition5/N-1
             Boot Path Retry:P1/N=3 P5/N-1=3  Recover
INFO: 00:000 15:265
Hit any key to stop autoboot:  0
WDT2: Disable in abortboot()
OSWDT: Disable in abortboot()
CPLD_BMCRDY: Enable BMC_MIN_RDY in abortboot().  Prevent BIOS reset.

NOTE: After stopping u-boot in this development mode.  You may need to
      warm/cold reset the server when booting iDRAC manually as BIOS
      may have already viewed iDRAC as unresponsive.



RECOVER:Max retries occured for both N/N-1 paths, OR forced recover.

 

Full Recovery Output

Sometimes it does not go well especially if you didn't know your EMMC is bad where it stores the firmware.

Not having the -noreset and -clear seems to have created this situation, with the waitpid but at that point the iDRAC lights and NIC were flashing.  I then hard powered down and had to use the DBG pins to get back into U-Boot.

iDRAC8=> util recover -emmc -from sd


UTIL RECOVER:Transport:sd  TargetMMC:EMMC  File:firmimg.d7
Loading file "firmimg.d7" from mmc device 1:1 (xxb1)
54542562 bytes read
UTIL RECOVER:SD load passed from EXT2 fs.
UTIL RECOVER:Transport time [sec:mil]: 46:594

Clear OS images in partition/s.
Clear kernelN, rootfsN, ubootN
'EMMC' blk size=[0x200][512]  Erase to 0xffffffff
Mem buf size=[0x06000000][100663296]  Total bytes=[0x08000000][134217728]
blocks[0x1:0x40001][1:262145]
Blocks [0x30001:0x40000] Buf[0x88000000]..fill buffer..erase/write
Completed loop 1,  Elapsed time in hr:min:sec:mil 14:122


Clear rootfsN1
'EMMC' blk size=[0x200][512]  Erase to 0xffffffff
Mem buf size=[0x06000000][100663296]  Total bytes=[0x06F80000][116916224]
blocks[0x48003:0x7fc03][294915:523267]
Blocks [0x78003:0x7fc02] Buf[0x88000000]..fill buffer..erase/write
Completed loop 1,  Elapsed time in hr:min:sec:mil 09:950
 
 
Clear ubootN1
'EMMC' blk size=[0x200][512]  Erase to 0xffffffff
Mem buf size=[0x06000000][100663296]  Total bytes=[0x00080000][524288]
blocks[0x7fc04:0x80004][523268:524292]
Blocks [0x7fc04:0x80003] Buf[0x88000000]..fill buffer..erase/write
Completed loop 1,  Elapsed time in hr:min:sec:mil 00:133
 

*** no text signature found ***

*** Updating Partition 1 ***
Checking image header CRC ... OK
Checking platform env ID..... OK
Checking kernel image CRC ... OK
Checking rootfs image CRC ... OK
Checking u-boot image CRC ... OK
Skipping u-boot update ...... NO
Checking Platform image CRC . OK
Checking processor support .. OK
Copying kernel image ........ OK
Copying rootfs .............. OK
Copying u-boot1 to flash..... OK
Copying u-boot2 to flash..... OK
Copying u-boot to MMC........ OK
Copying platform image ...... OK

*** Updating Partition 5 ***
Checking image header CRC ... OK
Checking platform env ID..... OK
Checking kernel image CRC ... OK
Checking rootfs image CRC ... OK
Checking u-boot image CRC ... OK
Skipping u-boot update ...... NO
Checking Platform image CRC . OK
Checking processor support .. OK
Copying kernel image ........ OK
Copying rootfs .............. OK
Copying u-boot1 to flash..... OK
Copying u-boot2 to flash..... OK
Copying u-boot to MMC........ OK
Copying platform image ...... OK
Erasing SPI flash at 0x100000...Writing to SPI flash...done
Erasing SPI flash at 0x110000...Writing to SPI flash...done
UTIL RECOVER:Program time [sec:mil]:01:14:394
UTIL RECOVER:Total update time [sec:mil]:02:00:992
Re-load 0x1000Kbytes from u-boot offset 0x0 into ILRAM at 0x0
Force Address Error to reset iDRAC
 
 
U-Boot 2009.08 (Mar 10 2012 - 02:52:09) Avocent (0.0.3) EVB, Build: jenkins-12G_iDRAC_Releases-402


CPU: SH-4A
BOARD: R0P7757LC00xxRL (C0 step) board
BOOT: Secure, HRK generated
DRAM:  240MB
  (240MB of 256MB total DRAM is available on U-Boot)
ENV:   Using primary env area.
In:    serial
Out:   serial
Err:   serial
WDT2:  Booted Lower Vector, 'uboot1'
sh_mmcif: 0, sh-sdhi: 1
Net:   sh_eth.0, sh_g_eth.0
INFO: 00:002 Start-up -to- util_idrac_main()
INFO: 00:005 U-Boot 2009.08 (Mar 10 2012 - 02:52:09) Avocent (0.0.3) EVB
INFO: 00:006 U-Boot checkin date(03-02-2012) Version(1.0.175)
INFO: 00:006 iDRAC PPID
INFO: 00:003 SPI NOR init 4096 KiB N25Q032 bus=0 cs=0, speed=1000000, mode=3
INFO: 00:007 SH-4A Product: Major Ver=0x31  Minor Ver=0x13 C3 Little endian
             Family=0x10    Major Ver=0x30  Minor Ver=0x0b
PASS: 00:016 Dedicated monolithic mgmt NIC disabled
INFO: 00:128 BCM54610 OUI=0x3fffff  Model=0x3f  Revision=0x0f PhyAddr=1
INFO: 00:365 SD CARD:   Device: sh-sdhi   Manufacturer ID: 41   OEM: 3432
             Name: DELL1   Tran Speed: 25000000   Rd Block Len: 512
             SD version 2.0   High Capacity: Yes   Capacity: -843055104
INFO: 00:058 EMMC:   Device: sh_mmcif   Manufacturer ID: 90   OEM: 14a
             Name: HYNIX   Tran Speed: 25000000   Rd Block Len: 512
             MMC version 4.0   High Capacity: Yes   Capacity: 0
INFO: 00:019 CPLD: Major Ver=0x1  Minor Ver=0x0  Maint Ver=0x3
             Planar: Type=0x02  Rev=0x8  Rework=0x0  Scratch/PathRetry=0x00
PASS: 00:014 Coin cell detected good,  AD=0x397 low water=0x2c1
PASS: 00:007 PCIe C0 Ver=0.13 MCTP en, CRC=0x19592849 @0x8efbf3a8 cnt=0x18000
INFO: 00:008 Init PCIe mailbox(PCIe 0xFFEE0150=0x40010000)
INFO: 00:005 mode=normal
INFO: 00:002 reset_cause=board
PASS: 00:005 Booted Lower Vector, 'uboot1' wdt2cnt=0
INFO: 00:005 wdt0cnt=0
PASS: 00:003 SMR0 no sermux env, default 0xd4
INFO: 00:004 GRACR=0x3c HISEL=0x00 SIRQCR5_D=0x03 SIRQCR6_D=0x01 LADMSK0=0xff2
             MRSTCR0=0xfedffe7f MRSTCR1=0xfff3ff0f MRSTCR2=0x7f80feff
             BARMAP=0x1 BCR=0x85000000 NCER=0x01fc NCMCR=0x0006 NCCSR=0x0303
PASS: 00:021 etherc0=78:45:C4:F6:88:68
             getherc0=78:45:C4:F6:88:69
INFO: 00:007 Fan logic is not modified on Non-AC power up
INFO: 00:053 Env and backup CRC'ed ok
Erasing SPI flash at 0x100000...Writing to SPI flash...done
Erasing SPI flash at 0x110000...Writing to SPI flash...done
*** no text signature found ***
INFO: 02:537 Sync eMMC/SPI NOR/Alternate u-boot images
PASS: 00:076 Current u-boot1 1.0.175 verified with 'ubootN'
             Trailer Struct - Missing start token, exp=0xc0de1111 rec=0x0
PASS: 03:612 Verify OS Images N: CRC32 OK: Kernel=0x4d21804a Rootfs=0x40007db7
PASS: 00:008 Boot device=emmc  Boot partition1/N
             Boot Path Retry:P1/N=0 P5/N-1=0
INFO: 00:000 06:990

WDT2: Disabled
Initialization complete, proceed with boot
*** no text signature found ***
 
No.  Device
---------------------
 1   EVB SD/MMC
 4   FLASH
 
Current Device ==> 1
 
OS_WDT0_EN:180 seconds,  Re-load ILRAM:
Boot Path Retry:P1/N=0 P5/N-1=0
Copying kernel image ... OK
## Booting kernel from Legacy Image at 82000000 ...
   Image Name:   sh-linux
   Image Type:   SuperH Linux Kernel Image (gzip compressed)
   Data Size:    2535377 Bytes =  2.4 MB
   Load Address: 80001000
   Entry Point:  80002000
   Verifying Checksum ... OK
   Uncompressing Kernel Image ... OK
WDT2: Disable in do_bootm_linux()
sh_eth_init
 
Monolithic/DRB
init started: BusyBox v1.18.4 (2012-03-10 03:35:27 UTC)
starting pid 415, tty '': '/etc/init.d/rcS'
waitfor: pid 415 ...
setting net.ipv6.conf.default.accept_dad based on kernel bootup option
enable normal accept DAD for ipv6
Sat Jan 1 00:00:01 UTC 2000 To execute command __source /etc/sysapps_script/I_1000_mountfs.sh__
Sat Jan 1 00:00:03 UTC 2000 To execute command __source /etc/sysapps_script/I_1001_trigwdt.sh__
<========= Start H/W watchdog first... =========>
Sat Jan 1 00:00:03 UTC 2000 To execute command __source /etc/sysapps_script/I_1100_nfs.sh__
Sat Jan 1 00:00:03 UTC 2000 To execute command __source /etc/sysapps_script/I_1105_ps.sh__
XXXXXX /dev/mmcblk0p11
/dev/mmcblk0p11: recovering journal
/dev/mmcblk0p11: Clearing orphaned inode 304 (uid=0, gid=0, mode=0100644, size=156)
/dev/mmcblk0p11: clean, 660/2048 files, 2831/4096 blocks (check in 5 mounts)
e2fsck 1.41.14 (22-Dec-2010)
/dev/mmcblk0p11: clean, 660/2048 files, 2831/4096 blocks (check in 5 mounts)
Sat Jan 1 00:00:04 UTC 2000 To execute command __source /etc/sysapps_script/I_1106_clear_ps.sh__
Sat Jan 1 00:00:04 UTC 2000 To execute command __source /etc/sysapps_script/I_1150_SYS_Drv.sh__
Sat Jan 1 00:00:07 UTC 2000 To execute command __source /etc/sysapps_script/I_1170_mknode.sh__
Sat Jan 1 00:00:08 UTC 2000 To execute command __/etc/sysapps_script/config_lib_init.sh__
mounting Platform Data 1 partition
Sat Jan 1 00:00:10 UTC 2000 To execute command __source /etc/sysapps_script/I_1160_OSINET.sh__
Sat Jan 1 00:00:10 UTC 2000 To execute command __source /etc/sysapps_script/I_1200_etc_defaults.sh__
Sat Jan 1 00:00:10 UTC 2000 To execute command __source /etc/sysapps_script/I_1210_sysklog_syslogd.sh__
starting syslogd
/etc/sysapps_script/syslog.sh: /sbin/aim not running
waitfor: pid 415, wpid 705
  syslogd: /mmc1/idraclogs: Read-only file system

Jan  1 00:00:11 (none) syslogd 1.4.2: restart.
Jan  1 00:00:12 (none) kernel: klogd 1.4.2, log source = /proc/kmsg started.
Jan  1 00:00:12 (none) kernel: Cannot find map file.
Jan  1 00:00:12 (none) kernel: No module symbols loaded - kernel modules not enabled.
Jan  1 00:00:12 (none) kernel: cannot find any symbols, turning off symbol lookups
Jan  1 00:00:12 (none) kernel: Linux version 2.6.30 (jenkins@vmbrd-dh-r5-253.us.dell.com) (gcc version 4.3.4 (GCC) ) #1 Sat Mar 10 02:59:38 UTC 2012
Jan  1 00:00:12 (none) kernel: Boot params:
Jan  1 00:00:12 (none) kernel: ... MOUNT_ROOT_RDONLY - 00000000
Jan  1 00:00:12 (none) kernel: ... RAMDISK_FLAGS     - 00000000
Jan  1 00:00:12 (none) kernel: ... ORIG_ROOT_DEV     - 00000000
Jan  1 00:00:12 (none) kernel: ... LOADER_TYPE       - 00000000
Jan  1 00:00:12 (none) kernel: ... INITRD_START      - 00000000
Jan  1 00:00:12 (none) kernel: ... INITRD_SIZE       - 00000000
Jan  1 00:00:12 (none) kernel: NMI Buffer at 0x83000000
Jan  1 00:00:12 (none) kernel: Booting machvec: SH7757LCR
Sat Jan 1 00:00:12 UTC 2000 To execute command __source /etc/sysapps_script/I_trab_asdf.sh__
Sat Jan 1 00:00:12 UTC 2000 To execute command __source /etc/sysapps_script/I_usb_drv.sh__
Jan  1 00:00:12 (none) kernel: Renesas R0P7757LC0012RL support.
Jan  1 00:00:12 (none) kernel: Node 0: start_pfn = 0x40000, low = 0x4ea00
Jan  1 00:00:12 (none) kernel: Zone PFN ranges:
Jan  1 00:00:12 (none) kernel:   Normal   0x00040000 -> 0x0004ea00
Jan  1 00:00:12 (none) kernel: Movable zone start PFN for each node
Jan  1 00:00:12 (none) kernel: early_node_map[1] activIRQ 50/r8a66597_udc: IRQF_DISABLED is not guaranteed on shared IRQs
e PFN ranges
JIRQ 57/r8a66597_udc: IRQF_DISABLED is not guaranteed on shared IRQs
an  1 00:00:12 (none) kernel:     0: 0x00040000 -> 0x0004ea00
Jan  1 00:00:12 (none) kernel: On node 0 totalpages: 59904
Jan  1 00:00:12 (none) kernel: free_area_init_node: node 0, pgdat 8048f3ac, node_mem_map 804f3000
Jan  1 00:00:12 (none) kernel:   Normal zone: 468 pages used for memmap
Jan  1 00:00:12 (none) kernel:   Normal zone: 0 pages reserved
Jan  1 00:00:12 (none) kernel:   Normal zone: 59436 pages, LIFO batch:15
Jan  1 00:00:12 (none) kernel: Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 59436
Jan  1 00:00:12 (none) kernel: Kernel command line: root=/dev/mmcblk0p2 rootwait rw rootfstype=squashfs mem=239616k console=ttyS2,115200 mac1=78:45:C4:F6:88:68 mac2=78:45:C4:F6:88:69  mode=normal reset_cause=board nmi_buf=0x83000000 quiet
Jan  1 00:00:12 (none) kernel: TRAB buffer at 8eb00000 allocated
Jan  1 00:00:12 (none) kernel: NR_IRQS:384
Jan  1 00:00:12 (none) kernel: PID hash table entries: 1024 (order: 10, 4096 bytes)
Jan  1 00:00:12 (none) kernel: Using tmu for system timer
Jan  1 00:00:12 (none) kernel: Dentry cache hash table entries: 32768 (order: 5, 131072 bytes)
Jan  1 00:00:12 (none) kernel: Inode-cache hash table entries: 16384 (order: 4, 65536 bytes)
Jan  1 00:00:12 (none) kernel: PVR=10300b00 CVR=73440410 PRR=00003113
Jan  1 00:00:12 (none) kernel: I-cache : n_ways=4 n_sets=256 way_incr=8192
Jan  1 00:00:12 (none) kernel: I-cache : entry_mask=0x00001fe0 alias_mask=0x00001000 n_aliases=2
Jan  1 00:00:12 (none) kernel: D-cache : n_ways=4 n_sets=256 way_incr=8192
Jan  1 00:00:12 (none) kernel: D-cache : entry_mask=0x00001fe0 alias_mask=0x00001000 n_ahid_init: Desc: Keyboard/Mouse Function
liases=2
Jan  hid_init: Name: g_kbdmouse
1 00:00:12 (nonehid_init: Version: 20110426
) kernel: Memoryhid_bind: Using Major Number 233
: 232192k/239616hid_bind: using r8a66597_udc, Keyboard ep1 Mouse ep2
k available (347hid_bind: Mouse REL ep3
0k kernel code, 1200k data, 156k init)
Jan  1 00:00:12 (none) kernel: Calibrating delay loop... 230.19 BogoMIPS (lpj=1150976)
Jan  1 00:00:12 (none) kernel: Security Framework initialized
Jan  1 00:00:12 (none) kernel: Mount-cache hash table entries: 512
Jan  1 00:00:12 (none) kernel: CPU: SH7757
Jan  1 00:00:12 (none) kernel: net_namespace: 980 bytes
Jan  1 00:00:12 (none) kernel: PCI: Using configuration type 1
Jan  1 00:00:12 (none) kernel: NET: Registered protocol family 16
Jan  1 00:00:12 (none) kernel: sh pinmux: sh7757_pfc handling gpio 0 -> 504
Jan  1 00:00:12 (none) kernel: bio: create slab at 0
Jan  1 00:00:12 (none) kernel: usbcore: registered new interface driver usbfs
Jan  1 00:00:12 (none) kernel: usbcore: registered new interface driver hub
Jan  1 00:00:12 (none) kernel: usbcore: registered new device driver usb
Jan  1 00:00:12 (none) kernel: DMA: Registering DMA API.
Jan  1 00:00:12 (none) kernel: DMA: Registering sh_dmac handler (24 channels).
Jan  1 00:00:12 (none) kernel: Autoconfig PCI channel 0x80488900
Jan  1 00:00:12 (none) kernel: Scanning bus 00, I/O 0xfed00000:0xff100000, Mem 0x04000000:0x08000000
Jan  1 00:00:12 (none) kernel: NET: Registered protocol family 2
Jan  1 00:00:12 (none) kernel: IP route cache hash table entries: 2048 (order: 1, 8192 bytes)
Jan  1 00:00:12 (none) kernel: TCP established hash table entries: 8192 (order: 4, 65536 bytes)
Jan  1 00:00:12 (none) kernel: TCP bind hash table entries: 8192 (order: 3, 32768 bytes)
Jan  1 00:00:12 (none) kernel: TCP: Hash tables configured (established 8192 bind 8192)
Jan  1 00:00:12 (none) kernel: TCP reno registered
Jan  1 00:00:12 (none) kernel: NET: Registered protocol family 1
Jan  1 00:00:12 (none) kernel: Switched to high resolution mode on CPU 0
Jan  1 00:00:12 (none) kernel: enable PCIe shared memory area
Jan  1 00:00:12 (none) kernel: audit: initializing netlink socket (disabled)
Jan  1 00:00:12 (none) kernel: type=2000 audit(946684800.310:1): initialized
Jan  1 00:00:12 (none) kernel: squashfs: version 4.0 (2009/01/31) Phillip Lougher
Jan  1 00:00:12 (none) kernel: JFFS2 version 2.2. (NAND) (SUMMARY)  © 2001-2006 Red Hat, Inc.
Jan  1 00:00:12 (none) kernel: fuse init (API version 7.11)
Jan  1 00:00:12 (none) kernel: msgmni has been set to 453
Jan  1 00:00:12 (none) kernel: alg: No test for cipher_null (cipher_null-generic)
Jan  1 00:00:12 (none) kernel: alg: No test for ecb(cipher_null) (ecb-cipher_null)
Jan  1 00:00:12 (none) kernel: alg: No test for digest_null (digest_null-generic)
Jan  1 00:00:12 (none) kernel: alg: No test for compress_null (compress_null-generic)
Jan  1 00:00:12 (none) kernel: alg: No test for stdrng (krng)
Jan  1 00:00:12 (none) kernel: io scheduler noop registered
Jan  1 00:00:12 (none) kernel: io scheduler anticipatory registered
Jan  1 00:00:12 (none) kernel: io scheduler deadline registered (default)
Jan  1 00:00:12 (none) kernel: io scheduler cfq registered
Jan  1 00:00:12 (none) kernel: Serial: 8250/16550 driver, 2 ports, IRQ sharing disabled
Jan  1 00:00:12 (none) kernel: SuperH SCI(F) driver initialized
Jan  1 00:00:12 (none) kernel: sh-sci: ttyS0 at MMIO 0xfe4b0000 (irq = 40) is a scif
Jan  1 00:00:12 (none) kernel: sh-sci: ttyS1 at MMIO 0xfe4c0000 (irq = 76) is a scif
Jan  1 00:00:12 (none) kernel: sh-sci: ttyS2 at MMIO 0xfe4d0000 (irq = 104) is a scif
Jan  1 00:00:12 (none) kernel: console [ttyS2] enabled
Jan  1 00:00:12 (none) kernel: sh-sci: ttyS3 at MMIO 0xfe4c0000 (irq = 76) is a scif
Jan  1 00:00:12 (none) kernel: sh-sci: ttyS4 at MMIO 0xfe4c0000 (irq = 76) is a scif
Jan  1 00:00:12 (none) kernel: sh-sci: ttyS5 at MMIO 0xfe4c0000 (irq = 76) is a scif
Jan  1 00:00:12 (none) kernel: sh-sci: ttyS6 at MMIO 0xfe4c0000 (irq = 76) is a scif
Jan  1 00:00:12 (none) kernel: brd: module loaded
Jan  1 00:00:12 (none) kernel: loop: module loaded
Jan  1 00:00:12 (none) kernel: nbd: registered device at major 43
Jan  1 00:00:12 (none) kernel: sh_eth_init
Jan  1 00:00:12 (none) kernel:
Jan  1 00:00:12 (none) kernel: Monolithic/DRB
Jan  1 00:00:12 (none) kernel: sh_mii: probed
Jan  1 00:00:12 (none) kernel: Base address at 0xfee00000, 00:00:00:00:00:00, IRQ 315.
Jan  1 00:00:12 (none) kernel: sh_mii: probed
Jan  1 00:00:12 (none) kernel: Base address at 0xfef00800, 00:00:00:00:00:00, IRQ 84.
Jan  1 00:00:12 (none) kernel: sh_spi sh_spi.0: registered master spi0
Jan  1 00:00:12 (none) kernel: m25p80 spi0.1: n25Q032a (4096 Kbytes)
Jan  1 00:00:12 (none) kernel: Creating 9 MTD partitions on "m25p80":
Jan  1 00:00:12 (none) kernel: 0x000000000000-0x000000080000 : "u-boot1"
Jan  1 00:00:12 (none) kernel: 0x000000080000-0x000000100000 : "u-boot2"
Jan  1 00:00:12 (none) kernel: 0x000000100000-0x000000110000 : "env1"
Jan  1 00:00:12 (none) kernel: 0x000000110000-0x000000120000 : "env2"
Jan  1 00:00:12 (none) kernel: 0x000000120000-0x000000130000 : "fru"
Jan  1 00:00:12 (none) kernel: 0x000000130000-0x000000140000 : "res1"
Jan  1 00:00:12 (none) kernel: 0x000000140000-0x0000001c0000 : "tracebuf"
Jan  1 00:00:12 (none) kernel: 0x0000001c0000-0x000000340000 : "lcl"
Jan  1 00:00:12 (none) kernel: 0x000000340000-0x000000400000 : "res2"
Jan  1 00:00:12 (none) kernel: sh_spi sh_spi.0: registered child spi0.1
Jan  1 00:00:12 (none) kernel: spi_sh: version 2009-06-10
Jan  1 00:00:12 (none) kernel: mknod /dev/aess_wdtdrv c 10 130
Jan  1 00:00:12 (none) kernel: aess_wdtdrv: initialized. (nowayout=0)
Jan  1 00:00:12 (none) kernel: device-mapper: ioctl: 4.14.0-ioctl (2008-04-23) initialised: dm-devel@redhat.com
Jan  1 00:00:12 (none) kernel: sh_mmcif: driver version 2010-03-17
Jan  1 00:00:12 (none) kernel: sh-sdhi: power control pin initiated
Jan  1 00:00:12 (none) kernel: mmc1: power off - 8f
Jan  1 00:00:12 (none) kernel: heartbeat: version 0.1.1 loaded
Jan  1 00:00:12 (none) kernel: sermux sermux: version 2010-05-26
Jan  1 00:00:12 (none) kernel: nf_conntrack version 0.5.0 (3744 buckets, 14976 max)
Jan  1 00:00:12 (none) kernel: IPv4 over IPv4 tunneling driver
Jan  1 00:00:12 (none) kernel: ip_tables: (C) 2000-2006 Netfilter Core Team
Jan  1 00:00:12 (none) kernel: TCP cubic registered
Jan  1 00:00:12 (none) kernel: NET: Registered protocol family 10
Jan  1 00:00:12 (none) kernel: lo: Disabled Privacy Extensions
Jan  1 00:00:12 (none) kernel: tunl0: Disabled Privacy Extensions
Jan  1 00:00:12 (none) kernel: ip6_tables: (C) 2000-2006 Netfilter Core Team
Jan  1 00:00:12 (none) kernel: IPv6 over IPv4 tunneling driver
Jan  1 00:00:12 (none) kernel: sit0: Disabled Privacy Extensions
Jan  1 00:00:12 (none) kernel: NET: Registered protocol family 17
Jan  1 00:00:12 (none) kernel: RPC: Registered udp transport module.
Jan  1 00:00:12 (none) kernel: RPC: Registered tcp transport module.
Jan  1 00:00:12 (none) kernel: 802.1Q VLAN Support v1.8 Ben Greear

This is mainly for if you've done something silly like trying to clone a Live, running VM image.  In this example, the VM initially finds grub and tries to boot but is kicked straight into initramfs rescue mode/busybox right after this.

If you've done this "silly" thing, you could have dataloss but a lot of times just using fsck will fix it as you are guaranteed at best to have some corruption and inconsistencies in the filesystem.  My theory is that some files were corrupt or not in their right places (eg. inode issues, unclaimed blocks etc..).

In many of these cases, just run an fsck and you may find that you can successfully boot, but how much data you lose is dependent on your application and how much the filesystem changed before you started the copy/vs when it ended.

The correct way is to take a snapshot or at least make sure the VM is off.  Another alternative if you can't shutdown is to do rsync or some other similar remote filecopy that constantly updates.

Many times just doing an update-grub may find Windows and add it to grub, but a lot of times it won't.

Create Windows in a custom grub entry like below:

sudo vi /etc/grub.d/40_custom

menuentry "Windows 10" {
   set root='(hd0,0)'
   chainloader + 1
}

Change Windows 10 to whatever you want to call it.  For example if it was for Windows 11 you'd probably want to call it "Windows 11" to avoid confusion.

The set root= part wil also vary.  If the drive that contains windows is known to hd0 as grub then keep it the same, but if you have other drives, it could be hd1,hd2 etc... and depending on your partition table setup the second part after the "," 0 means partition 1 but sometimes it may be msdos1.

Here's an example of booting Windows from the second hd on msdos3 partiton AKA partition #3

menuentry "Windows 10" {
   set root='(hd0,msdos3)'
   chainloader + 1
}

Now update grub otherwise the added menu entry won't show or be available when you boot grub:

sudo update-grub

This guide assumes you have a working Postfix server and want it to sign with DKIM.

There are a few things we  have to understand to make all of this work though, which require you to be familiar with DNS as well.

1.) Install OpenDKIM

apt install opendkim

systemctl enable opendkim

2.) Edit /etc/opendkim.conf

Syslog   yes
SyslogSuccess yes
Mode     sv
OmitHeaders .
Socket   inet:8891@localhost
Domain   yourdomain.com
KeyTable        /etc/opendkim/KeyTable
SigningTable   refile:/etc/opendkim/SigningTable
ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
InternalHosts   refile:/etc/opendkim/TrustedHosts
UserID  opendkim:opendkim


3.) Create your first OpenDKIM Key for your domain

yourdomain=yourdomain.com


mkdir -p /etc/opendkim/keys/$yourdomain
cd /etc/opendkim/keys/$yourdomain
opendkim-genkey -r -d $yourdomain


4.) Configure KeyTable and SigningTable

#Add this line to /etc/opendkim/KeyTable

The "default" is what is called the selector, it could be nearly anything but we're calling it default here.  Be sure to change "yourdomain.com" to the actual domain you want DKIM signing for.

default._domainkey.yourdomain.com yourdomain.com:default:/etc/opendkim/keys/yourdomain.com/default.private

#Add this to line to /etc/opendkim/SigningTable

*@yourdomain.com default._domainkey.yourdomain.com

The * allows all e-mails from the domain to be signed, change it if this should not be the case.

Again, be sure you change yourdomain.com to your actual domain that you setup in the KeyTable in the previous step.

*Remember to restart opendkim after adding any new domain or the change will not be applied and e-mails will not be signed.

5.) Edit your Your DNS zonefile for yourname.com
 

Here's an example using bind/named:

#edit bind/named DNS entry for domain/zone

cat /etc/opendkim/keys/yourdomain.com/default.txt

Take the output of the above and then add it to your zonefile like below:

default._domainkey    IN    TXT    ( "v=DKIM1; k=rsa; s=email; "
      "p=blablabla" )  ; ----- DKIM key default for yourdomain.com

Be sure to change the blablabla to the output of /etc/opendkim/keys/yourdomain.com/default.txt

_dmarc.yourdomain.com.      IN     TXT    "v=DMARC1; p=quarantine; rua=mailto:someaddress@yourdomain.com"
 

Be sure to change someaddress@yourdomain.com to a real address for your domain, you will receive DMARC reports to this address.

Make sure your zonefile has a valid SPF record or none of this will work properly.

yourdomain.com.    IN TXT "v=spf1 a mx +all"

The above is a quick example of a zonefile entry in bind for SPF, it allows your domain's A record MX record to send mail and doesn't have a hard fail.  If for example you need to add another server's IP (perhaps another server needs to send mail for yourdomain.com) then update as below:

As you'll see adding the +ip4:8.8.8.8 would allow a server with 8.8.8.8 to send mail for the domain.

yourdomain.com.    IN TXT "v=spf1 a mx +ip4:8.8.8.8 +all"

 

Set permissions to be sure:

chown -R opendkim.opendkim /etc/opendkim

5.) Enable OpenDKIM milter in Postfix

# edit /etc/postfix/main.cf
cp -a /etc/postfix/main.cf /etc/postfix/main.cf-`date +%Y%m%d-%s`
 

#edit /etc/postfix/main.cf

#enable dkim
smtpd_milters           = inet:127.0.0.1:8891
non_smtpd_milters       = $smtpd_milters
milter_default_action   = accept

systemctl restart postfix

 *Remember to restart opendkim after adding any new domain or the change will not be applied and e-mails will not be signed.

apt install tftpd-hpa

#change TFTP_ADDRESS to by setting address to 192.168.1.1:69 or the IP you need, otherwise it will listen on all IPs and interfaces which could be a security risk.

# edit /etc/default/tftpd-hpa

TFTP_USERNAME="tftp"
TFTP_DIRECTORY="/srv/tftp"
TFTP_ADDRESS="192.168.1.1:69"
TFTP_OPTIONS="--secure"

systemctl restart tftpd-hpa

#now edit our isc dhcp server /etc/dhcp/dhcpd.conf

#under the subnet declaration:

subnet 192.168.1.0 netmask 255.255.255.0 {
  range 192.168.1.2 192.168.1.200;
  #deny unknown-clients;
  option routers 192.168.1.1;
  option domain-name-servers 192.168.1.1;
  next-server 192.168.1.1;
  filename "pxelinux.0";
}

#how to exclude hosts from receiving a DHCP
# add this in the subnet declaration
  host whatevernameyouwant {
   hardware ethernet 00:10:28:e2:49:2e;
   ignore booting;
  }
#comment out the ignore booting and restart dhcpd if you want to re-enable it
 

###
apt install syslinux pxelinux
#for MBR copy this

cp /usr/lib/PXELINUX/pxelinux.0 /srv/tftp/
cp /usr/lib/syslinux/modules/bios/ldlinux.c32 /srv/tftp/
cp /usr/lib/syslinux//modules/bios/menu.c32 /srv/tftp/
cp /usr/lib/syslinux//modules/bios/memdisk /srv/tftp/
cp /usr/lib/syslinux//modules/bios/mboot.c32 /srv/tftp/
cp /usr/lib/syslinux//modules/bios/chain.c32 /srv/tftp/
cp /usr/lib/syslinux//modules/bios/libutil.c32 /srv/tftp/

mkdir /srv/tftp/images
mkdir /srv/tftp/pxelinux.cfg

vi /srv/tftp/pxelinux.cfg/default

default menu.c32
prompt 5
timeout 300

MENU TITLE Menu
LABEL rtt-Diagnostic
MENU rtt-Diagnostic
kernel ../images/kernelfs

service isc-dhcp-server restart
 

This sometimes happens when trying to install the EFI version of grub to a device when you are booted into Legacy/MBR mode.  It doesn't seem to occur on all machines, but some and seems somewhat BIOS dependent.

grub-install --target=x86_64-efi /dev/sda
Installing for x86_64-efi platform.
grub-install.real: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
grub-install.real: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
EFI variables are not supported on this system.
efibootmgr: option requires an argument -- 'd'
efibootmgr version 15
usage: efibootmgr [options]
    -a | --active         sets bootnum active
    -A | --inactive       sets bootnum inactive
    -b | --bootnum XXXX   modify BootXXXX (hex)
    -B | --delete-bootnum delete bootnum
    -c | --create         create new variable bootnum and add to bootorder
    -C | --create-only    create new variable bootnum and do not add to bootorder
    -D | --remove-dups    remove duplicate values from BootOrder
    -d | --disk disk       (defaults to /dev/sda) containing loader
    -r | --driver         Operate on Driver variables, not Boot Variables.
    -e | --edd [1|3|-1]   force EDD 1.0 or 3.0 creation variables, or guess
    -E | --device num      EDD 1.0 device number (defaults to 0x80)
    -g | --gpt            force disk with invalid PMBR to be treated as GPT
    -i | --iface name     create a netboot entry for the named interface
    -l | --loader name     (defaults to "EFIubuntugrub.efi")
    -L | --label label     Boot manager display label (defaults to "Linux")
    -m | --mirror-below-4G t|f mirror memory below 4GB
    -M | --mirror-above-4G X percentage memory to mirror above 4GB
    -n | --bootnext XXXX   set BootNext to XXXX (hex)
    -N | --delete-bootnext delete BootNext
    -o | --bootorder XXXX,YYYY,ZZZZ,...     explicitly set BootOrder (hex)
    -O | --delete-bootorder delete BootOrder
    -p | --part part        (defaults to 1) containing loader
    -q | --quiet            be quiet
    -t | --timeout seconds  set boot manager timeout waiting for user input.
    -T | --delete-timeout   delete Timeout.
    -u | --unicode | --UCS-2  handle extra args as UCS-2 (default is ASCII)
    -v | --verbose          print additional information
    -V | --version          return version and exit
    -w | --write-signature  write unique sig to MBR if needed
    -y | --sysprep          Operate on SysPrep variables, not Boot Variables.
    -@ | --append-binary-args file  append extra args from file (use "-" for stdin)
    -h | --help             show help/usage
grub-install.real: error: efibootmgr failed to register the boot entry: Operation not permitted.



[Wed Nov 01 18:47:08 2023] [error] Unable to configure verify locations for client authentication
[Wed Nov 01 18:47:08 2023] [error] SSL Library Error: 151441510 error:0906D066:PEM routines:PEM_read_bio:bad end line
[Wed Nov 01 18:47:08 2023] [error] SSL Library Error: 185090057 error:0B084009:x509 certificate routines:X509_load_cert_crl_file:PEM lib

It actually gives us a good clue that at last one component of our cert is invalid/improperly formatted.  So check your CA Bundle, Cert and Key to make sure they are all correct.  After that, restart apache2 and everything should be good

This can break things easily in remove environments where it was normally easy to convert a normal eth0 to a bridge under br0, and that bridge would normally have the same MAC address by default, which is desirable for most situations.

In Debian 11 this is different for some reason now.

https://unix.stackexchange.com/questions/681013/bridge-gets-random-mac-instead-of-port-address

One simple solution is to set the hwaddress in /etc/network/interfaces:

iface br0 inet static
bridge_ports eth0
    address 172.16.1.3
    netmask 255.255.255.0
    gateway 172.16.1.1
    hwaddress 00:1d:e0:00:13:58
 

Set the hwaddress to the mac address of eth0 to solve this problem.  What happens is that if you don't do this, the server will become inaccessible after some random amount of time, at least initially until a reboot or network restart.

This may be a result of poor cloning without regenerating the machine-id:

#possibly related, we need to delete the machine-id and generate a new one
rm /var/lib/dbus/machine-id
rm /etc/machine-id

Regenerate new machine-id:

The file lives in /etc and /var/lib/dbus and both must match.

dbus-uuidgen --ensure=/etc/machine-id


cp /etc/machine-id /var/lib/dbus/

sysctl vm.overcommit_memory=1

echo never > /sys/kernel/mm/transparent_hugepage/enabled
echo 511 > /proc/sys/net/core/somaxconn

1:M 26 Nov 2023 21:34:33.840 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 26 Nov 2023 21:34:33.840 # Server initialized
1:M 26 Nov 2023 21:34:33.840 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 26 Nov 2023 21:34:33.840 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 26 Nov 2023 21:34:33.840 * Ready to accept connections


Assign way more replicas than you have of memory on all nodes and watch the Swarm crash which can easily reproduce in a small VM for testing.

root@Deb11Docker01:~# docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED        STATUS        PORTS      NAMES
1336c9b0275e   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.7.ue8ff3g3ef2pv4ezu4cy5qc23
f5496728a590   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.11.kk3ij52i7dq0p4xqe3r05apzc
b0b2a47dc6b6   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.8.p5woskaqpyc6kod0njfmdq5m9
6dc2c2a0f8f2   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.23.x27zfpun3mf8j40qofers8mqs
91e264dc4ec9   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.17.lma6im540lw77uk6nolxg7rej
47830106e26d   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.18.8y6rm833sgxvbtnj1np4e8q0i
a4246ca036e9   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.19.z7jj4hiz6sjyux2hmgy6d9449
965754f35798   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.13.ue6b58bkx4w44n8gkht0bu0r8
d4a9572cae71   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.6.4sa0179qxpoy4m522d5mc2shy
1ea0cd1e493c   redis:5   "docker-entrypoint.s…"   13 hours ago   Up 13 hours   6379/tcp   clever_bouman.28.nmzn1jd77lybe211thm1f9q8n
root@Deb11Docker01:~# docker service update --replicas=60 clever_bouman
clever_bouman
overall progress: 51 out of 60 tasks
Error response from daemon: rpc error: code = DeadlineExceeded desc = context deadline exceeded
root@Deb11Docker01:~# docker service update --replicas=40 clever_bouman
Error response from daemon: rpc error: code = Unknown desc = The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online.

[137927.550974] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=docker.service,mems_allowed=0,global_oom,task_memcg=/system.slice/docker.service,task=dockerd,pid=55994,uid=0
[137927.551065] Out of memory: Killed process 55994 (dockerd) total-vm:1859264kB, anon-rss:80588kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:664kB oom_score_adj:0



root@Deb11Docker02:~# ps aux|grep container
root        1404  0.0  3.5 1370844 35244 ?       Ssl  Nov30   1:44 /usr/bin/containerd
root       48601  1.3  0.3 1328876 3528 ?        Sl   02:34   0:08 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 8be1aac07725843e36d47934209ee280ec2705df3bd098a9320eb0abd659eaaa -address /run/containerd/containerd.sock
root       49954  0.9  0.4 1329132 4456 ?        Sl   02:35   0:05 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 3445ae9397be1d9cbb80dcf9679c7765c2e6f5093672543b4c16d5b0e7935c71 -address /run/containerd/containerd.sock
root       50004  0.9  1.4 1322088 14560 ?       Sl   02:35   0:05 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 72301e658829c8fc3679b0744a110100cd31423c6f510bf330e8b90842c308cb -address /run/containerd/containerd.sock
root       50037  0.7  0.6 1314148 6360 ?        Sl   02:35   0:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 13f22679e833befe833f14073d800d6fa6a04e44bb30e317dc01a282b19a3e1b -address /run/containerd/containerd.sock
root       50045  0.7  0.4 1322088 4560 ?        Sl   02:35   0:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 1c0e7f3e65032590ac3078afc2b227c1003680c7b8c48c755193a27b17027aec -address /run/containerd/containerd.sock
root       50061  0.7  0.6 1328940 6472 ?        Sl   02:35   0:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id b258bac1daebc0eebc75195802f22631991b99effda0c8dd32d916dda53b3ecd -address /run/containerd/containerd.sock
root       50109  0.7  0.4 1322088 4672 ?        Sl   02:35   0:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id 5d9b13fc6a53e71a9b0efcf9524b4b410843a6351b273020cb38bd49fdeda8d1 -address /run/containerd/containerd.sock
root       50150  1.0  0.6 1320680 6564 ?        Sl   02:35   0:05 /usr/bin/containerd-shim-runc-v2 -namespace moby -id cb18a243b3cc30d4d62d9fec8995c14da81e8c2a3ed6de670a880a4e174c4e55 -address /run/containerd/containerd.sock
root       51985  0.0  0.0   6180   716 pts/0    S+   02:44   0:00 grep container
root@Deb11Docker02:~# killall -9 containerd-shim-runc-v2
root@Deb11Docker02:~# ps aux|grep container
root        1404  0.0  3.5 1379040 35244 ?       Ssl  Nov30   1:44 /usr/bin/containerd
root       52092  0.0  0.0   6180   716 pts/0    S+   02:45   0:00 grep container
root@Deb11Docker02:~#


 docker swarm init --force-new-cluster

The error below can be caused by a gateway that is unpingable:

docker swarm join --token SWMTKN-1-1kogg8da68gtb1j7ezaddowyy9s0an5s9tue758o20k18liskw-5h3f61hrrmv3u6agshvbtcklf 172.16.1.80:2377
Error response from daemon: manager stopped: can't initialize raft node: rpc error: code = Unknown desc = could not connect to prospective new cluster member using its advertised address: rpc error: code = DeadlineExceeded desc = context deadline exceeded

Nov 28 02:01:52 dockertest01 dockerd[14612]: time="2023-11-28T02:01:52.189193519-05:00" level=warning msg="Failed to dial 172.16.1.1:2377: grpc: the connection is closing; please retry." module=grpc

The error below doesn't make sense at first, you join a certain IP but the error comes back as trying to connect to a different IP. 

This can be caused my forwarding and / or proxies which should be disabled to fix the problem.

Even with pingable gateway it now wants to somehow connect on port 2377 through the gateway instead of the .80 manager node IP?

 docker swarm join --token SWMTKN-1-1kogg8da68gtb1j7ezaddowyy9s0an5s9tue758o20k18liskw-5h3f61hrrmv3u6agshvbtcklf 172.16.1.80:2377
Error response from daemon: manager stopped: can't initialize raft node: rpc error: code = Unknown desc = could not connect to prospective new cluster member using its advertised address: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp 172.16.1.1:2377: connect: connection refused"


You can do a static lease that is tied to the MAC address but what a lot of users prefer is that they come into the office or lab the next day and that their device gets assigned the same IP address (if possible).

As we can see in the dhcpd logs that there is threshold that is defaulted as we'll show later.  Whatever the threshold is set at, if the lease is younger than the threshold, it will keep the same lease.  In other words, if the device goes to sleep or is powered off for too long, it would be more likely to get a new IP.

dhcpd[2408362]: reuse_lease: lease age 138 (secs) under 25% threshold, reply with unaltered, existing lease for 172.17.1.7
 

This is all controlled in dhcpd.conf

The default lease time is 10 minutes and maximum 120 minutes.  This is clearly not sufficient for user's who may be offline for several hours or a few days.

#default-lease-time 600;
#max-lease-time 7200;

Some suggest putting a -1 which is infinite, as the number which is not recommended but MAY be OK in smaller settings.  In larger settings you would want some sort of finite time even if it's x amount of days.

default-lease-time -1;
max-lease-time -1;
 

This seems to happen in many different drivers but it happened more often in newer versions such as 530 vs 525.

Then nvidia-modeset goes to 100%

There are many reports of this appearing since driver 4.70 and I can confirm I've seen this in various machines.

https://forums.developer.nvidia.com/t/black-screen-when-resuming-systemctl-suspend-using-nvidia-driver-470-57-02-with-kernel-5-8-0-63-generic-on-gtx-970-xubuntu-20-04-lts/184644/57

https://forums.developer.nvidia.com/t/not-coming-back-from-suspend/176446
https://forums.developer.nvidia.com/t/systemds-suspend-then-hibernate-not-working-in-nvidia-optimus-laptop/213690
https://forums.developer.nvidia.com/t/black-screen-when-resuming-systemctl-suspend-using-nvidia-driver-470-57-02-with-kernel-5-8-0-63-generic-on-gtx-970-xubuntu-20-04-lts/184644
 

This report proposes a fix that work for some people:

https://forums.developer.nvidia.com/t/fixed-suspend-resume-issues-with-the-driver-version-470/187150

Basically the idea is to disable all nvidia related systemd services:

systemctl stop nvidia-suspend
systemctl stop nvidia-hibernate
systemctl stop nvidia-resume

systemctl disable nvidia-suspend
systemctl disable nvidia-hibernate
systemctl disable nvidia-resume

#let's just copy it and save it to ~ just in case
mv /lib/systemd/system-sleep/nvidia ~

Here is the dmesg output when this happens, the system wakes up but the GPU hangs:

[245728.273127] WARNING: CPU: 12 PID: 25855 at /var/lib/dkms/nvidia/525.105.17/build/nvidia/nv.c:3913 nv_restore_user_channels+0x12f/0x1e0 [nvidia]
[245728.273128] Modules linked in: bluetooth ecdh_generic ecc uas usb_storage ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs cpuid snd_seq_dummy xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter aufs xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp bridge stp llc iptable_filter bpfilter overlay nls_iso8859_1 binfmt_misc nvidia_uvm(OE) nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) intel_rapl_msr intel_rapl_common input_leds sb_edac x86_pkg_temp_thermal intel_powerclamp kvm_intel snd_hda_codec_hdmi kvm snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul crc32_pclmul ledtrig_audio ghash_clmulni_intel aesni_intel snd_hda_intel snd_intel_dspcfg crypto_simd drm_kms_helper snd_hda_codec cryptd snd_hda_core drm glue_helper snd_hwdep fb_sys_fops snd_pcm syscopyarea snd_seq_midi snd_seq_midi_event sysfillrect snd_rawmidi rapl sysimgblt snd_seq intel_cstate mei_me dell_smbios snd_seq_device
[245728.273154]  dcdbas snd_timer dell_smm_hwmon snd wmi_bmof intel_wmi_thunderbolt dell_wmi_descriptor mei soundcore lpc_ich mac_hid mxm_wmi sch_fq_codel hwmon_vid coretemp parport_pc ppdev lp parport ip_tables x_tables autofs4 hid_generic usbhid hid btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear dm_mirror dm_region_hash dm_log raid1 igb e1000e dca i2c_algo_bit ahci libahci wmi
[245728.273172] CPU: 12 PID: 25855 Comm: nvidia-sleep.sh Tainted: P           OE     5.4.0-149-generic #166~18.04.1-Ubuntu
[245728.273172] Hardware name: rtt
[245728.273293] RIP: 0010:nv_restore_user_channels+0x12f/0x1e0 [nvidia]
[245728.273294] Code: 44 89 f8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 4c 89 ef 45 31 ff e8 42 8d 89 d5 44 89 f8 5b 41 5c 41 5d 41 5e 41 5f 5d c3 45 31 f6 <0f> 0b 4c 89 e7 e8 d7 8c 89 d5 be 01 00 00 00 48 89 df e8 0a a3 00
[245728.273295] RSP: 0018:ffffac3d849d7dc8 EFLAGS: 00010206
[245728.273296] RAX: 0000000000000003 RBX: ffff91f0c026d000 RCX: ffffac3d849d7d48
[245728.273296] RDX: ffffac3d80b93e50 RSI: 0000000000000246 RDI: ffffac3d849d7cf8
[245728.273297] RBP: ffffac3d849d7df0 R08: 0000000000000000 R09: 0000000000000001
[245728.273297] R10: ffffac3d849d7aa0 R11: 00000000000084ca R12: ffff91f0c026d678
[245728.273298] R13: ffff91f0c026d550 R14: ffff91eefdd13000 R15: 0000000000000003
[245728.273299] FS:  00007f6323bf9740(0000) GS:ffff91f0cf900000(0000) knlGS:0000000000000000
[245728.273299] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[245728.273300] CR2: 0000557b4901f008 CR3: 000000041d754004 CR4: 00000000003606e0
[245728.273300] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[245728.273301] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[245728.273301] Call Trace:
[245728.273378]  nv_set_system_power_state+0x2d7/0x480 [nvidia]
[245728.273455]  nv_procfs_write_suspend+0xe3/0x160 [nvidia]
[245728.273458]  proc_reg_write+0x3e/0x60
[245728.273460]  __vfs_write+0x1b/0x40
[245728.273461]  vfs_write+0xb1/0x1a0
[245728.273462]  ksys_write+0xa7/0xe0
[245728.273463]  __x64_sys_write+0x1a/0x20
[245728.273465]  do_syscall_64+0x57/0x190
[245728.273468]  entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[245728.273469] RIP: 0033:0x7f63232f7104
[245728.273470] Code: 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 8d 05 e1 08 2e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 49 89 d4 53 48 89 f5
[245728.273471] RSP: 002b:00007ffc609a56c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[245728.273472] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007f63232f7104
[245728.273472] RDX: 0000000000000007 RSI: 0000557b4901f700 RDI: 0000000000000001
[245728.273473] RBP: 0000557b4901f700 R08: 000000000000000a R09: 0000000000000006
[245728.273473] R10: 000000000000000a R11: 0000000000000246 R12: 00007f63235d3760
[245728.273474] R13: 0000000000000007 R14: 00007f63235cf2a0 R15: 00007f63235ce760
[245728.273475] ---[ end trace 2ff2aa81ae8de839 ]---
[245728.273484] ------------[ cut here ]------------
[245728.273559] WARNING: CPU: 12 PID: 25855 at /var/lib/dkms/nvidia/525.105.17/build/nvidia/nv.c:4143 nv_set_system_power_state+0x31f/0x480 [nvidia]
[245728.273560] Modules linked in: bluetooth ecdh_generic ecc uas usb_storage ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs cpuid snd_seq_dummy xt_conntrack nf_conntrack_netlink nfnetlink xfrm_user xfrm_algo xt_addrtype br_netfilter aufs xt_CHECKSUM iptable_mangle xt_MASQUERADE iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xt_tcpudp bridge stp llc iptable_filter bpfilter overlay nls_iso8859_1 binfmt_misc nvidia_uvm(OE) nvidia_drm(POE) nvidia_modeset(POE) nvidia(POE) intel_rapl_msr intel_rapl_common input_leds sb_edac x86_pkg_temp_thermal intel_powerclamp kvm_intel snd_hda_codec_hdmi kvm snd_hda_codec_realtek snd_hda_codec_generic crct10dif_pclmul crc32_pclmul ledtrig_audio ghash_clmulni_intel aesni_intel snd_hda_intel snd_intel_dspcfg crypto_simd drm_kms_helper snd_hda_codec cryptd snd_hda_core drm glue_helper snd_hwdep fb_sys_fops snd_pcm syscopyarea snd_seq_midi snd_seq_midi_event sysfillrect snd_rawmidi rapl sysimgblt snd_seq intel_cstate mei_me dell_smbios snd_seq_device
[245728.273574]  dcdbas snd_timer dell_smm_hwmon snd wmi_bmof intel_wmi_thunderbolt dell_wmi_descriptor mei soundcore lpc_ich mac_hid mxm_wmi sch_fq_codel hwmon_vid coretemp parport_pc ppdev lp parport ip_tables x_tables autofs4 hid_generic usbhid hid btrfs zstd_compress raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid0 multipath linear dm_mirror dm_region_hash dm_log raid1 igb e1000e dca i2c_algo_bit ahci libahci wmi
[245728.273583] CPU: 12 PID: 25855 Comm: nvidia-sleep.sh Tainted: P        W  OE     5.4.0-149-generic #166~18.04.1-Ubuntu
[245728.273583] Hardware name: rtt
[245728.273664] RIP: 0010:nv_set_system_power_state+0x31f/0x480 [nvidia]
[245728.273665] Code: ff e8 f5 e8 00 00 48 c7 c7 20 73 df c3 e8 e9 5e 89 d5 89 1d e7 9c 38 03 e9 4a fd ff ff 4c 89 f7 e8 96 59 89 d5 e9 49 fe ff ff <0f> 0b eb b8 45 31 ff 48 c7 c7 d0 72 df c3 45 31 e4 e8 7b 59 89 d5
[245728.273666] RSP: 0018:ffffac3d849d7e00 EFLAGS: 00010206
[245728.273667] RAX: 0000000000000003 RBX: 0000000000000002 RCX: 000000000001d4c0
[245728.273667] RDX: 000000000001d4bf RSI: 9380d93cdab66952 RDI: 00003a4cb021fac0
[245728.273668] RBP: ffffac3d849d7e30 R08: 0000000000000000 R09: 0000000000000001
[245728.273668] R10: ffffac3d849d7aa0 R11: 00000000000084ca R12: ffff91f0c026d000
[245728.273669] R13: 0000000000000000 R14: 0000557b4901f700 R15: ffff91ede71f6800
[245728.273669] FS:  00007f6323bf9740(0000) GS:ffff91f0cf900000(0000) knlGS:0000000000000000
[245728.273670] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[245728.273670] CR2: 0000557b4901f008 CR3: 000000041d754004 CR4: 00000000003606e0
[245728.273671] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[245728.273671] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[245728.273672] Call Trace:
[245728.273748]  nv_procfs_write_suspend+0xe3/0x160 [nvidia]
[245728.273750]  proc_reg_write+0x3e/0x60
[245728.273752]  __vfs_write+0x1b/0x40
[245728.273753]  vfs_write+0xb1/0x1a0
[245728.273754]  ksys_write+0xa7/0xe0
[245728.273755]  __x64_sys_write+0x1a/0x20
[245728.273756]  do_syscall_64+0x57/0x190
[245728.273758]  entry_SYSCALL_64_after_hwframe+0x5c/0xc1
[245728.273758] RIP: 0033:0x7f63232f7104
[245728.273759] Code: 89 02 48 c7 c0 ff ff ff ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 8d 05 e1 08 2e 00 8b 00 85 c0 75 13 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 f3 c3 66 90 41 54 55 49 89 d4 53 48 89 f5
[245728.273760] RSP: 002b:00007ffc609a56c8 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[245728.273760] RAX: ffffffffffffffda RBX: 0000000000000007 RCX: 00007f63232f7104
[245728.273761] RDX: 0000000000000007 RSI: 0000557b4901f700 RDI: 0000000000000001
[245728.273761] RBP: 0000557b4901f700 R08: 000000000000000a R09: 0000000000000006
[245728.273762] R10: 000000000000000a R11: 0000000000000246 R12: 00007f63235d3760
[245728.273762] R13: 0000000000000007 R14: 00007f63235cf2a0 R15: 00007f63235ce760
[245728.273763] ---[ end trace 2ff2aa81ae8de83a ]---
[245731.275215] nvidia-modeset: WARNING: GPU:0: Lost display notification (0:0x00000000); continuing.
[245733.531067] nvidia-modeset: ERROR: GPU:0: Idling display engine timed out: 0x0000947d:0:0:407

 

Sometimes users take their removal drives and unplug and replug them to test what happens during the failure of a disk.  However, this breaks things quite badly due to the /dev/mapper in LUKS not coming back online due to it not being closed.

In other words, generally with non-encrypted drives the process is smooth but when encrypted you may want to follow a strategy like this:

We can see below that both disks are unavailable as they were physically removed from the server.

zpool status

  pool: rttpool
 state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: http://zfsonlinux.org/msg/ZFS-8000-HC
  scan: none requested
config:

    NAME            STATE     READ WRITE CKSUM
    rttpool         UNAVAIL      0     0     0  insufficient replicas
      mirror-0      UNAVAIL      0     0     0  insufficient replicas
        zpool-sdj1  FAULTED      0     0     0  corrupted data
        zpool-sdk1  FAULTED      0     0     0  corrupted data
errors: List of errors unavailable: pool I/O is currently suspended

Conventional wisdom says to clear the error after replugging the disks but does this work with LUKS?

root@rttbox:/home/rtt# zpool clear rttpool zpool-sdj1
cannot clear errors for zpool-sdj1: I/O error
root@rttbox:/home/rtt# zpool clear rttpool zpool-sdj
cannot clear errors for zpool-sdj: no such device in pool
root@rttbox:/home/rtt# zpool clear rttpool zpool-sdj1
cannot clear errors for zpool-sdj1: I/O error
root@rttbox:/home/rtt# zpool clear rttpool zpool-sdk1
cannot clear errors for zpool-sdk1: I/O error

As we can see, no it doesn't work.

Sometimes we may need to remove zpool.cache

#at your own risk do not try in production or not as a first resort just in case

rm /etc/zfs/zpool.cache

Now let's force clear the pool

 zpool clear -nF rttpool
root@rttbox:/home/rtt# zpool status
  pool: rttpool
 state: UNAVAIL
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: http://zfsonlinux.org/msg/ZFS-8000-HC
  scan: none requested
config:

    NAME            STATE     READ WRITE CKSUM
    rttpool         UNAVAIL      0     0     0  insufficient replicas
      mirror-0      UNAVAIL      0     0     0  insufficient replicas
        zpool-sdj1  FAULTED      0     0     0  too many errors
        zpool-sdk1  FAULTED      0     0     0  too many errors
errors: List of errors unavailable: pool I/O is currently suspended


It still doesn't work as we can see.

How about clearing the device in the pool itself?

root@rttbox:/home/rtt# zpool clear -nF rttpool zpool-sdj1
root@rttbox:/home/rtt# zpool clear -nF rttpool zpool-sdk1


root@rttbox:/home/rtt# zpool online rttpool zpool-sdk1
cannot online zpool-sdk1: pool I/O is currently suspended
root@rttbox:/home/rtt# zpool online rttpool zpool-sdj1
cannot online zpool-sdj1: pool I/O is currently suspended

We can see that it still doesn't fix it.

The Actual Solution

Properly use cryptsetup to close and remove the zpool devices.

cryptsetup close zpool-sdj1
cryptsetup close zpool-sdk1

Now reopen the devices:

cryptsetup open /dev/sdj1 zpool-sdj1
cryptsetup open /dev/sdk1 zpool-sdk1

#then do cryptsetup open
zpool clear -nFX rttpool

now it works!


 zpool status
  pool: rttpool
 state: ONLINE
  scan: resilvered 160K in 0h0m with 0 errors on Thu Feb  1 23:01:59 2024
config:

    NAME            STATE     READ WRITE CKSUM
    rttpool         ONLINE       0     0     0
      mirror-0      ONLINE       0     0     0
        zpool-sdj1  ONLINE       0     0     0
        zpool-sdk1  ONLINE       0     0     0

errors: No known data errors

How To Recover After Moving ZFS Disks to new computer/server

If your disks are ready to go, zpool import will scan all disks looking for ZFS.

zpool import
   pool: rttpool
     id: 125324434212034535323
  state: ONLINE
 action: The pool can be imported using its name or numeric identifier.
 config:

    rttpool         ONLINE
      mirror-0      ONLINE
        zpool-sdc1  ONLINE
        zpool-sdd1  ONLINE
 

Now just import it by the numeric ID 125324434212034535323 or the pool name rttpool

zpool import 125324434212034535323

Have you got this error from Apache?

[notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[error] (28)No space left on device: Cannot create SSLMutex

At first glance it appears that you may be out of disk space but the issue is ipc or interprocess communication.

This will clear out the ipcs processes so things can work, this often happens during high traffic and may be a sign of DDOS.

The command below will fix it, it will list all of the ipcs processes that belong to your apache user (change the part in bold to whatever apache runs as) and then

ipcs -s | grep YOURAPACHEuser | awk '{ print $2 }' | xargs -n 1 ipcrm -s

The above should fix it, but if you're curious here is the raw output of ipcs -s on a random VM.

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 0          root       600        1         
0x00000000 65537      root       600        1         
0x00000000 1081346    apache     600        1         
0x00000000 1114115    apache     600        1  

That's why we do awk on the second column to get the semid, and then we pipe to xargs which then uses the ipcrm -s command to remove all of the semid's that we found.

Even today we see a lot of servers that have different services and ports open for rpc and this creates not only potential inward vulnerabilities but perhaps more common, the abuse of your network resources in reflective rpc queries.

To stop this problem, you should disable and remove all services relating to rpc or at least block all relevant ports for the service.

Surprisingly, there are still some providers and OS installs in Linux that install these services and leave them open.  I've seen it cost companies up to thousands per month or more based on RPC being open and available which is then used in reflective DDOS.

Disable these services relating to RPC to stop the abuse:

rpcbind
rpcgssd

One common thing is that you may find servers with unexpectedly higher traffic than normal that cannot be attributed to the main applications running.  It may be less obvious if it's a slight bump in traffic on already busy nodes, so look carefully and ensure your base images/containers are locked down.

Have you ever tried mounting a partition that you exists but you get this error?

mount: /mnt: can't read superblock on /dev/sda1.

The superblock in this example was bad because the physical disk had corruption and bad blocks/sectors.  However, the data was generally accessible and you can always try this trick below (with caution and no warranty).

This is specifically for filesystems that place superblocks in multiple locations, which makes the drive accessible.

Run the mkfs for your filesystem, in this case it was ext4 and pass the "-n" option. 

mkfs.ext3 -n /dev/sda1

*Do not forget the -n above or you will destroy the partition.

Then you'll see the prompt below, once again make sure you used -n

       -n     Causes  mke2fs  to not actually create a filesystem, but display
              what it would do if it were to create a filesystem.  This can be
              used  to  determine the location of the backup superblocks for a
              particular filesystem, so long as  the  mke2fs  parameters  that
              were  passed when the filesystem was originally created are used
              again.  (With the -n option added, of course!)
 

Hit y to proceed as long as you're sure that you used the -n flag.

mke2fs 1.44.1 (24-Mar-2018)
/dev/sda1 contains a ext4 file system
    last mounted on / on Wed Jun  8 11:40:24 2022
Proceed anyway? (y,N) y

Then you'll hopefully see a list of superblocks as below:

ext2fs_check_if_mount: Can't check if filesystem is mounted due to missing mtab file while determining whether /dev/sda1 is mounted.
Creating filesystem with 120038912 4k blocks and 30015488 inodes
Filesystem UUID: 91fcf16a-7f97-47cf-82b3-654d7dca6d70
Superblock backups stored on blocks:
    32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
    4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
    102400000


You can try any of those locations, if the disk is badly damaged you may need to try a few different superblocks.

Here is how you would mount with one of the superblocks above.

mount -o sb=4096000 /dev/sda1 /mnt
 

The -o sb= is where we plugin one of the superblocks we located earlier, and I used one of them from 4096000. 

Hopefully you can access and recover most of your data if this happens to you.

convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `/tmp/magick-27777Al6FGY7dhyKt10' @ error/cache.c/OpenPixelCache/3984.
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `/tmp/magick-277772Y_-pJnMdT2r1' @ error/cache.c/OpenPixelCache/3984.
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `/tmp/magick-27777m6KQRhWOtkWp1' @ error/cache.c/OpenPixelCache/3984.
convert-im6.q16: DistributedPixelCache '127.0.0.1' @ error/distribute-cache.c/ConnectPixelCacheServer/244.
convert-im6.q16: cache resources exhausted `/tmp/magick-27777ERzzOPUYQMc11' @ error/cache.c/OpenPixelCache/3984.

 

You just have to increase the limits in policy.xml to solve the issue:

vi /etc/ImageMagick-6/policy.xml

Set values like this (increase as necessary):


<policy domain="resource" name="area" value="6GiB"/>
<policy domain="resource" name="disk" value="6GiB"/>
<policy domain="resource" name="memory" value="4GiB"/>
<policy domain="resource" name="map" value="4GiB"/>

Be sure not to set levels that are bigger than your disk size and available RAM.

PTY allocation request failed on channel 0

You may have messed up your environment and specifically under /dev you may have remounted a wrong source point or otherwise killed /dev/pts

The solution is to recreate /dev/pts or fix whatever caused it to be wiped out or broken.

In verbose mode the client may show this:


#verbose client
Authenticated to 172.16.17.2 ([172.16.17.2]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network

Nov 15 17:00:49 rttbox kernel: overlayfs: filesystem on '/var/lib/docker/overlay2/check-overlayfs-support450709549/upper' not supported as upperdir
Nov 15 17:00:49 rttbox dockerd[93755]: failed to start daemon: error initializing graphdriver: driver not supported


The above error is often/normally caused because you are trying to run docker out of an unsuitable directory/filesystem.  For example if you try to place docker's data inside an existing overlayfs2 location, then you will get this error.  You cannot put another overlayfs2 layer from docker over top of an already existing overlayfs2.

This is a weird issue as sometimes when upgrading or even migrating, this could happen and the reason is simple but maybe not 100% obvious at first. 

You will find that your GUI doesn't load and most services fail to start, even logind

Here are some errors you may see:


Mar 13 22:22:23 rttbox systemd-logind[2892]: Failed to connect to system bus: No such file or directory
Mar 13 22:22:23 rttbox systemd-logind[2892]: Failed to fully start up daemon: No such file or directory
Mar 13 22:22:24 rttbox systemd-logind[3795]: Failed to connect to system bus: No such file or directory

Mar 13 22:22:47 rttbox dbus-daemon[2282]: [system] Failed to activate service 'org.freedesktop.ColorManager': timed out (service_start_timeout=25000ms)
Mar 13 22:22:47 rttbox dbus-daemon[2282]: [system] Failed to activate service 'org.freedesktop.systemd1': timed out (service_start_timeout=25000ms)
Mar 13 22:22:47 rttbox dbus-daemon[2282]: [system] Failed to activate service 'org.freedesktop.PolicyKit1': timed out (service_start_timeout=25000ms)
Mar 13 22:22:48 rttbox dbus-daemon[2282]: [system] Failed to activate service

The easiest way to know if this is your cause is to check /var/run:

ls -al /var/run
lrwxrwxrwx 1 root root 4 Jul  2  2019 /var/run -> /run


/var/run should be a symlink to /run, if it's not then you'll need to fix it like this:

First we copy any relevant contents of /var/run into /run, then we delete /var/run and resymlink it to /run as it should be.

cp -a /var/run/* /run/
rm -rf /var/run
ln -s /run /var/run

Now make /run/dbus in case it's not there and restart the dbus service:

mkdir /run/dbus
systemctl restart dbus

After this everything should be OK, you may need to reboot if it's still not.

You start a qemu like this and another VM already started with the same .iso

qemu-system-x86_64 -enable-kvm -m 4096 -cdrom some.iso

But you get this error:

qemu-system-x86_64: Initialization of device ide-hd failed: Failed to get "write" lock
Is another process using the image [some.iso]?

It's odd because the -cdrom by default is read-only and qemu should not want or care about getting write lock on the .iso.

What causes this lock issue with an iso?

qemu-system-x86_64 -enable-kvm -m 4096 some.iso

Note that the .iso above was used, it will work and boot but it was passed to QEMU the same way as a normal disk image, so therefore it puts a write lock on it, even though it doesn't need to be.

Solution Use The ISO as a -cdrom

Make sure that you use -cdrom and then you can start unlimited amounts of VMs using that image.

qemu-system-x86_64 -enable-kvm -m 4096 -cdrom some.iso

Is python3-pip pip3 not working anymore?

Traceback (most recent call last):
  File "/usr/bin/pip3", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.5/dist-packages/pip/__init__.py", line 11, in main
    from pip._internal.utils.entrypoints import _wrapper
  File "/usr/local/lib/python3.5/dist-packages/pip/_internal/utils/entrypoints.py", line 12
    f"pip{sys.version_info.major}",
                                 ^
SyntaxError: invalid syntax


Sometimes this is caused by a mismatch in versions, perhaps your python3-pip came from a different PPA/source so it no longer matches your installed version.

Make sure that all tools come from the same source or if you need a new python version, that the entirety including all helper tools are compiled from the same source.

You may have been expecting a normal sized kernel but if you don't disable these .config options you may have a kernel that is several gigabytes.


CONFIG_SLUB_DEBUG=no


CONFIG_DEBUG_INFO=no

CONFIG_DEBUG_MISC=no

Rebuild and you should find that the kernel is the normal and as expected size.

[    0.206301] [Firmware Bug]: TSC_DEADLINE disabled due to Errata; please update microcode to version: 0x3a (or later)
[    0.430409] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[    0.430411] MMIO Stale Data: Vulnerable: Clear CPU buffers attempted, no microcode
[    2.980359] microcode: sig=0x306f2, pf=0x1, revision=0x36
[    2.981621] microcode: Microcode Update Driver: v2.2.


 

If you get an error like this just install the microcode package for your architecture:

For Intel CPUs do this:
apt install intel-microcode

For AMD CPUs do this:

apt install amd64-microcode

Are you getting this error when trying to connect to a site/domain/service and then you checked your router/nameserver's logs to see what's wrong?

You may see something like this:

named[3025898]: REFUSED unexpected RCODE resolving

This is not usually an error with named or bind, at least not on your end but NORMALLY this is an issue with the remote nameserver.

A lot of times this is a misconfigured remote nameserver.  A classic case may be that the remote nameserver is up but the zonefile is non-existent or misconfigured or even hacked.

You should contact the domain admin e-mail of the remote side to see what is happening.

Some people find it less than intuitive to do on DNSMasq and by default DNSMasq is available on 0.0.0.0 which could even be your LAN or Public IP.

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:53              0.0.0.0:*               LISTEN      2139/dnsmasq   

How do we fix it so DNSMasq listens on localhost only?

The default that most will do is edit /etc/dnsmasq.conf and then set this

listen-address=127.0.0.1

But all that is needed is actuallly the "bind-interfaces" option and in fact if you don't have that option, it will still listen on 0.0.0.0 which may not be what you want.

Note that even interface= will not help, especially if you don't have the bind-interfaces option as we show below.

So just add this to /etc/dnsmasq.conf

bind-interfaces



Then restart dnsmasq and check:

ctive Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:53            0.0.0.0:*               LISTEN      2230/dnsmasq        

If you have du, you may want to check your bill, as you can read about a longstanding issue with fraudulent charges showing up and many users claiming they did not subscribe or solicit those offers. 

These don't normally show up on new accounts, but they seem to target established users and maybe even users they suspect are not watching their phone or bills, while they are on vacation.

These charges can frighteningly happen with 0 interaction from the user despite what du will say.

https://www.reddit.com/r/dubai/comments/6o7y0t/hey_du_read_this/

https://www.reddit.com/r/dubai/comments/w57njs/du_subscribed_me_to_3rd_party_subscriptions/

Have you noticed strange charges to your bill even though you didn't have your phone on at the time of the supposed charges and subscriptions?

To du's credit, they normally reverse the charges if you complain, especially if you offer proof that you didn't click any links or send an SMS to authorize the billing.

Login to du and then check your plan and "VAS" and make sure you are not subscribed to any services that you didn't authorize:

Be warned that even though VAS may show no subscriptions, you can and will still be impossibly charged for subscriptions that you haven't subscribed to.

Conclusion

Be vigilant, even if du tells you that everything has been canceled and refunded, I have found that this is normally NOT the case and will require followups, so be sure to take screenshots and then complain to the TDRA.

A lot of companies are unsure which solution to choose and many may not be aware of Docker Swarm as an alternative to Kubernetes.  One thing that many Sysadmins find is that Docker Swarm is simply easier, quicker to setup and maintain by far than Kubernetes. 

See for yourself with this Docker Tutorial which includes Docker Swarm.

See the difference with one of our favorite and most efficient Kubernetes implementations.

One striking feature is that if we compare deploying the same container by setting up a 3 node Docker Swarm or Kubernetes, it is MUCH quicker to get the same application going in Kubernetes.

Administration and Expertise

• Docker Swarm: Given its simplicity and integration with Docker, the administration overhead for Docker Swarm is significantly lower than Kubernetes. It requires less specialized knowledge, making it easier for teams already familiar with Docker to manage a Swarm environment. This translates into lower costs related to training and hiring specialized personnel. Smaller teams can effectively manage Swarm clusters without the need for dedicated DevOps roles focused solely on cluster management.

• Kubernetes: Kubernetes' complexity and rich feature set necessitate a higher level of expertise and more administration time. Managing a Kubernetes cluster involves understanding various components like pods, services, deployments, and more, which has a steeper learning curve. Organizations often need to invest in specialized training for their teams or hire experienced Kubernetes administrators. This can significantly increase the manpower cost. Additionally, the ongoing management and optimization of Kubernetes clusters require more dedicated resources, potentially leading to higher costs in terms of both manpower and administration time.

Operational Costs

• Docker Swarm: The operational costs of running Docker Swarm are generally lower, not just in terms of the computing resources required but also considering the manpower needed for its administration. Its lightweight nature means that it can run on less powerful hardware or cloud instances, further reducing costs.

• Kubernetes: Kubernetes can be more resource-intensive, requiring more robust hardware or higher-tier cloud services to accommodate its operational demands. Additionally, the cost of employing skilled Kubernetes operators or DevOps engineers can be substantial. While Kubernetes can optimize resource utilization and cost through features like auto-scaling, the initial setup, ongoing management, and optimization require continuous effort and expertise.

Total Cost of Ownership (TCO)

The Total Cost of Ownership for Kubernetes vs. Docker Swarm extends beyond just the immediate resource and manpower costs. It includes training, the complexity of operations, the scale of deployment, and the potential need for additional tools or services to manage and monitor the environment.

• Docker Swarm: Offers a lower TCO for smaller to medium-sized deployments or for teams with limited DevOps resources. Its ease of use and lower resource requirements make it a cost-effective solution for many scenarios, especially when operational simplicity and lower manpower costs are prioritized.

• Kubernetes: While the TCO can be higher, especially with larger deployments requiring skilled personnel, Kubernetes' scalability and efficiency can, in the long run, offer cost savings for complex, large-scale applications and environments that benefit from its advanced features. The investment in Kubernetes can lead to better resource utilization, higher availability, and more dynamic scaling, potentially offsetting the higher initial costs over time.

Ease of Setup and Use

• Docker Swarm: It is significantly easier to set up and configure Docker Swarm compared to Kubernetes. This is because Docker Swarm is tightly integrated into the Docker ecosystem. Setting up a Swarm cluster can be as simple as initiating a few Docker commands. This simplicity extends to its management and operation, making it more accessible for beginners or teams with smaller-scale deployments.

• Kubernetes: Setting up a Kubernetes cluster is more complex and involves a steeper learning curve. Kubernetes offers more features and flexibility, which comes at the cost of increased complexity in setup and management. However, tools like Minikube or managed Kubernetes services (e.g., EKS, AKS, GKE) can help alleviate some of the setup difficulties.

Scalability

• Docker Swarm: Offers fast and straightforward scalability. Swarm is designed to be efficient in scaling up and down in response to demand. However, it might not handle extremely large clusters as efficiently as Kubernetes.

• Kubernetes: Known for its robust scalability features. It can manage much larger clusters efficiently and provides more options for high availability and load balancing. Kubernetes is the go-to for enterprises needing to scale their applications massively.

Features and Flexibility

• Docker Swarm: Provides a simpler model for deploying applications, with a focus on ease of use and integration with Docker containers. It lacks some of the advanced features found in Kubernetes but covers the basic needs of container orchestration well.

• Kubernetes: Offers a rich set of features including but not limited to auto-scaling, self-healing, service discovery, and load balancing. Its extensibility and flexibility make it suitable for complex applications and workflows.

Resource Requirements

• Docker Swarm: Requires fewer resources compared to Kubernetes, making it a more economical option for smaller operations or where resources are limited. Its lightweight nature means it can run effectively on smaller setups.

• Kubernetes: Due to its extensive feature set and the complexity of managing large clusters, Kubernetes generally requires more resources (CPU, memory) to run effectively. This can be a consideration for organizations with limited resources.

Community and Ecosystem

• Docker Swarm: While Docker Swarm enjoys support from the Docker community, its ecosystem is not as vast or active as Kubernetes'. This can affect the availability of third-party tools and integrations.

• Kubernetes: Boasts a large and active community. The ecosystem around Kubernetes is rich with third-party tools, services, and integrations, making it easier to find support and resources for extending Kubernetes capabilities.

Use Cases

• Docker Swarm: Ideal for small to medium-sized deployments, startups, or for those who prefer simplicity and are already invested in the Docker ecosystem. It's also well-suited for applications that do not require the complex orchestration features offered by Kubernetes.

• Kubernetes: Suited for larger, more complex applications and enterprises requiring high scalability, extensive automation, and advanced deployment strategies. It's the choice for multi-cloud and hybrid-cloud environments due to its flexibility and wide industry support.

Disaster Recovery Implications

1. Faster Initial Recovery: Given that Docker Swarm allows for quicker deployment, in the event of a disaster, restoring services to operational status can be achieved more rapidly. This speed is beneficial for minimizing downtime and reducing the impact on business operations.

2. Ease of Management: Docker Swarm's simplicity also means it's easier to manage in stressful situations, such as during disaster recovery. Fewer complexities reduce the risk of errors during the recovery process, making it more straightforward to execute recovery plans.

3. Scalability Concerns: While Docker Swarm offers speed and simplicity, it's important to note that Kubernetes provides more advanced features for handling large-scale, complex applications. In disaster recovery, this might mean Kubernetes can offer better long-term scalability and resilience options, despite its slower initial setup time. However, for many applications, Docker Swarm's capabilities are more than adequate, especially when rapid recovery is a priority.

4. Automation and Orchestration: Kubernetes excels in automation and orchestration, which can be advantageous in automatically managing application deployment and recovery across a large number of nodes. Docker Swarm, while simpler, may require more manual intervention in complex scenarios. However, for many scenarios, the speed and ease of use of Docker Swarm can outweigh these benefits, especially in environments where the application architecture aligns well with Swarm's model.

Can Docker Swarm Be Used at Scale?

Large Deployments Using Docker Swarm

Companies have chosen Docker Swarm for its lightweight nature and ease of use, especially when their infrastructure does not demand the extensive scalability and complex orchestration features provided by Kubernetes. Companies in sectors like technology, finance, and e-commerce have utilized Docker Swarm for hosting web applications, microservices, and internal tools, appreciating its straightforward scaling and deployment mechanisms.

Limitations and Scalability

• Nodes: Docker Swarm is designed to be scalable and handle a large number of nodes efficiently. Officially, Docker has tested and supports Swarm clusters up to 2,000 nodes and 100,000 containers with no significant impact on performance.   Keep in mind this was back in 2016 and there are no known or listed limitations in how many nodes Docker can have.  These numbers serve as a benchmark for Docker Swarm's scalability, indicating its capability to support large-scale deployments. However, the practical limit can depend on the specific architecture, network setup, and the workload of the applications being run. https://etoews.github.io/blog/2016/08/05/joining-100-nodes-to-a-docker-swarm/

• Practical Considerations: While Docker Swarm can technically support thousands of nodes, the practical scalability limit for a cluster might be influenced by factors like network latency, cluster management overhead, and the complexity of deployed services. For most use cases, Docker Swarm offers sufficient scalability. It's also worth noting that very large-scale applications that require intricate orchestration, advanced scheduling, or extensive automation might reach a point where Kubernetes offers advantages due to its richer feature set and ecosystem.

• Examples of large deployments of Docker Swarm: There are reports of loads runnning 100,000 containers and more than 10,000 nodes.  There is no known limit that we've seen, so it is untrue that Docker Swarm cannot be deployed at scale.

• More than 100 Mirantis customers utilize Swarm for production workloads, including GlaxoSmithKline, MetLife, Royal Bank of Canada, and S&P Global. This translates to more than 10,000 nodes spread across approximately 1,000 clusters, supporting over 100,000 containers orchestrated by Swarm.

Conclusion

Choosing between Docker Swarm and Kubernetes depends on your project's size, complexity, and specific requirements. Docker Swarm is a great choice for those valuing simplicity, ease of setup, and lower resource requirements. Kubernetes, on the other hand, is suited for those needing a highly scalable, feature-rich platform capable of managing complex applications and workflows. Each tool has its strengths, and the decision should align with your technical needs, team's expertise, and long-term strategy.  The easiest answer to the question is that if you don't need the extra features that Kubernetes provides, then you cannot go wrong with Docker Swarm.

In today’s digital landscape, finding a reliable and secure Virtual Private Server (VPS) or Virtual Dedicated Server (VDS) goes beyond just comparing specs and prices. With increasing concerns over data privacy, security breaches, and government surveillance, the wisdom of choosing your VPS/VDS provider based on jurisdiction, security features, and operational transparency has never been more critical.

Our guide on when to switch to a different provider is here.

Why is our VPS guide different?

Many of the blogs and guides out there are not made by professionals with decades of experience, and in fact, are often created by full time digital marketing gurus and firms.

realtechtalk.com is a source of technical information and solutions, many of the other guides out there and comparisons are created by "Tech blogs" that don't actually run mission critical operations or real world problems and solutions.

The sheer amount of available guides can be overwhelming. However, it's crucial to approach these resources with a discerning eye, especially considering that many guides are driven by underlying motives such as affiliate marketing, prioritizing recommendations that may not align with your best interests. Our guide stands apart in this cluttered space, and here's why it's an invaluable resource for businesses prioritizing security and reliability.

Assessing Technical Depth and Utility

1. Detailed How-To Guides and Tutorials: Genuine technical blogs often provide comprehensive how-to guides, tutorials, and step-by-step instructions that help readers solve specific problems or achieve tasks. The presence of detailed, actionable content can indicate a blog's commitment to offering real value to its audience.

2. Problem-Solving Content: Authentic tech blogs usually address common and niche technical issues, offering solutions based on experience and expertise. They might discuss troubleshooting steps, optimization tips, and best practices for using VPS and cloud services effectively.

3. Technical Discussion Depth: Blogs that delve into the technical nuances, underlying principles, and architectural considerations of VPS and cloud solutions demonstrate a deeper understanding of the subject matter. A focus on surface-level features or benefits without discussing the technical how and why may suggest a lack of genuine expertise.

Unbiased and Comprehensive Analysis

1. No Affiliate Links: Unlike many guides that rely on affiliate marketing for revenue, our recommendations are untainted by financial incentives. This means we have no hidden agenda; our sole focus is to provide you with the best options based on merit, not commission rates.

2. Security-Centric Recommendations: We understand that for businesses, security isn't just another checkbox—it's a foundation. Our guide is crafted with a deep understanding of cybersecurity threats and countermeasures, ensuring that our recommendations prioritize the safety of your data above all else.

3. Reliability as a Priority: In the digital age, downtime is synonymous with lost revenue and damaged reputation. We highlight solutions known for their uptime, robust infrastructure, and excellent customer support, ensuring your operations run smoothly around the clock.

Prioritize Long-Standing Providers

• Established Track Record: Opt for a provider that has been in business for a significant period. Longevity in the hosting industry often indicates a consistent track record of reliability and customer satisfaction. Companies that have weathered the ups and downs of the market are more likely to understand the nuances of maintaining high uptime, offering scalable solutions, and providing responsive customer support.

Assess Ownership and Management Experience

• Reputable Ownership: Consider providers that are either independently reputable or are part of a well-regarded parent company known for its stability and financial health. The backing of a reputable company often means access to better resources, infrastructure, and a more significant safety net in terms of business continuity and disaster recovery capabilities.

• Experienced Management: Look into the experience and background of the management team. Providers led by individuals with a proven track record in the IT, datacenter, or networking industries can offer more reliable and innovative services. Experienced leadership is adept at navigating technical challenges and market fluctuations, ensuring the provider remains on solid footing.

Verify the Provider's Financial Health

• Stable Financial Background: A financially stable provider is less likely to experience service disruptions due to budget constraints or go out of business suddenly, jeopardizing your data and online presence. While financial details may not always be publicly available, signs of stability include consistent service improvements, infrastructure investments, and positive customer testimonials over the years.

Look for Consistent Innovation and Improvement

• Ongoing Investment in Technology: A provider that continually invests in its infrastructure, software, and services is likely in a stable financial position and committed to offering the best possible service. Regular updates to services, hardware, and security protocols indicate a provider's dedication to maintaining high performance and customer satisfaction.

Choose Lesser-Known, Specialized Providers

• Avoid Household Names: Opt for providers that may not be household names but have a solid reputation in specialized forums and industry circles. These companies are less likely to be the target of mass-scale cyber attacks and may offer more personalized service, which can be a boon for SMEs requiring attentive support.

Evaluate the Provider's Infrastructure and Expertise

• Assess Scale and Reliability: While large providers may offer attractive pricing due to their scale, their service can sometimes be impersonal, and they may experience significant outages. A smaller, more specialized provider could offer more reliable uptime and personalized attention, critical for businesses where every minute of downtime counts.

• Verify In-House Expertise: Ensure the provider has in-house experts in datacenter management, servers, and networking. Providers that openly share information about their team's expertise and infrastructure are generally more trustworthy and capable of delivering higher service quality. Those that don't may lack direct control over their services, potentially compromising reliability and performance.

Look for Comprehensive Services and Support

• Full-Scale System and Network Support: Choose a provider that offers a range of support and consulting services, from system administration to network security. This indicates a higher level of expertise and a commitment to supporting your business's specific needs. Providers that control their own hardware and network infrastructure are preferable, as they can offer more customized solutions and faster response times.

Importance of Control and Transparency

• Control Over Hardware and Network: A provider that owns and controls its hardware and network infrastructure can offer better security, performance, and reliability. This control also means they can be more responsive to issues and offer more customized solutions, essential for businesses with specific or changing needs.

Why Security and Jurisdiction Matter

Recent years have shown that even the most reputable providers are not immune to outages, security breaches, and other vulnerabilities. High-profile incidents involving major companies like Amazon, Vultr, and DigitalOcean have highlighted the risks of hosting with providers that become prime targets for cyberattacks. The allure of free or cheap services often comes with a hidden cost: your data’s security and privacy. Many of these low-cost providers have been identified as sources of hacking traffic, making them not just insecure but complicit in the broader ecosystem of cyber threats.

Moreover, the revelations from Edward Snowden about NSA surveillance activities have cast a long shadow over providers in PRISM countries. Providers under the jurisdiction of these nations are subject to laws that compel them to hand over user data to government agencies, often without the user's knowledge or consent. This reality poses a significant risk for businesses and individuals concerned about the confidentiality of their communications and the integrity of their data.

Signs of Expertise, Does Your Provider Offer Dedicated Servers, Fiber and IP Space?

Comprehensive Infrastructure Solutions

• Private Server Capabilities: Look for providers that offer the option to transition from VPS to dedicated or even private server environments. This capability is crucial for businesses with growing or fluctuating demands, ensuring that they can scale their resources up or down as needed without compromising performance or security.

Dedicated Network Options

• Custom Network Solutions: Providers that can offer dedicated network solutions, including private VLANs or MPLS, indicate a high level of network management expertise. This is essential for businesses that prioritize data security and require isolated networking environments to protect sensitive information.

Private Fiber Optic Connections

• Direct Fiber Access: A provider capable of offering direct fiber optic connections for your business' exclusive use, demonstrates a significant investment in infrastructure. This level of service ensures the highest possible speed and reliability, and security, crucial for businesses that rely on real-time data access and high-bandwidth applications.

Dedicated IP Space

• Allocated IP Ranges: Providers that can offer dedicated IP space give businesses the flexibility and control needed for various operations, including running their own web servers, email servers, and other services while ensuring improved security and reputation management.

Expertise and Customization

• Security and Scalability: The ability to provide these high-level services is a testament to a provider's expertise in secure and scalable IT services. It indicates a depth of knowledge not only in hardware and infrastructure but also in understanding the unique needs of businesses that require these advanced services.

Avoid Free Trials and Money Back Guarantee VPS/VDS/Cloud Offers

In addition to the concerns surrounding large VPS providers, it's also wise to exercise caution with providers that offer free trials or 30-day money-back guarantees. While these offers are attractive for legitimate users testing the services before committing financially, they present vulnerabilities that can be exploited by malicious actors. Below, we explore why such promotional offers can exacerbate the risks of shared hosting environments.

Increased Susceptibility to Abuse by Hackers

1. Low Barriers to Entry: Free trials and money-back guarantees lower the barriers to entry, making it easier for hackers to sign up for services with minimal upfront investment. This accessibility allows malicious users to exploit these platforms for launching attacks, hosting malicious content, or conducting other nefarious activities without significant financial risk.

2. Temporary Nature of Services: The temporary nature of free or trial accounts makes them particularly appealing to hackers. Malicious actors can use these services to perform their activities and then abandon the account before any serious action can be taken against them. This transient usage pattern complicates efforts to track and mitigate abusive behavior.

3. Reputation Damage and Blacklisting: As hackers frequently utilize these trial accounts, the IP ranges and resources allocated to these promotional offers can become associated with malicious activities. This association can lead to IPs being blacklisted, affecting not only the hackers but also legitimate users who are later assigned those IPs or subnets. The consequences can include email delivery issues, decreased network reputation, and potential blocking by security systems.

4. Resource Strain and Security Risks: Providers offering extensive free trials or guarantees might attract a disproportionate number of abusers, straining resources and potentially compromising the security of the network. This strain can impact service quality for legitimate users and increase the provider's susceptibility to breaches if resources are diverted to manage abuse instead of enhancing security measures.

Mitigation Strategies for Users

To mitigate these risks, users should consider providers that have robust verification processes for new accounts, even if they offer trials or money-back guarantees. Additionally, researching a provider's reputation, security practices, and how they handle abuse can provide insights into their suitability. Opting for providers that balance promotional offers with strict anti-abuse measures can help minimize the risk of being inadvertently associated with malicious activities.

Avoid Large Providers

The choice of a Virtual Private Server (VPS) provider is crucial for businesses and individuals who prioritize privacy, security, and reliability in their online operations. Opting for large VPS providers, while seemingly advantageous due to their established reputations and extensive resources, comes with a set of risks that warrant careful consideration. Below are key reasons why one might avoid large VPS providers, focusing on concerns related to mass surveillance, government interest, hacker targeting, and the implications of sharing IP space with malicious actors.

Increased Risk of Mass Surveillance and Government Interest

1. Scale and Visibility: Large VPS providers, due to their size and the volume of data they handle, are more likely to be under surveillance by governments and intelligence agencies. The extensive customer base and significant amount of traffic make these providers prime targets for mass data collection efforts.

2. Legal and Regulatory Compliance: Big providers are often subject to stringent legal and regulatory requirements, which may compel them to comply with government requests for data access. In jurisdictions with laws that infringe on privacy rights, this compliance can lead to the monitoring of user activities without explicit consent or knowledge.

3. Data Center Locations: The global presence of large providers means that some of their data centers may be located in countries with aggressive surveillance laws. This geographic diversity, while beneficial for performance and redundancy, can expose users to varied legal regimes, some of which may be more intrusive.

Vulnerability to Hacking and Malicious Activities

1. Attractive Targets for Hackers: The prominence and scale of large VPS providers make them attractive targets for cybercriminals and hackers. Breaching the defenses of a major provider can grant attackers access to a vast amount of resources and data, offering a high return on their efforts.

2. Shared IP Space Concerns: When you use a large VPS provider, you share IP space with numerous other customers, including potentially malicious users. This shared environment can lead to your IP addresses being blacklisted or marked on Real-time Blackhole Lists (RBLs) due to the activities of other users, affecting your email deliverability and reputation.

3. Proximity to Malicious Actors: Sharing network infrastructure with hackers increases the risk of collateral damage from targeted attacks against other users on the same network. Moreover, if hackers manage to exploit vulnerabilities within the provider's network, there's a theoretical risk they could gain unauthorized access to your resources, especially if network segmentation or isolation measures are inadequate.

Choosing a VPS/VDS Provider: Features to Look For

When searching for a VPS/VDS provider, prioritize those registered and operating outside of PRISM countries. Look for jurisdictions with strong data protection laws and a history of resisting extraterritorial demands for user data.

Security Features

• Data Encryption: Ensure that the provider offers robust encryption for data at rest and in transit.
• DDoS Protection: Look for built-in protection against Distributed Denial of Service (DDoS) attacks.
• Regular Security Audits: Choose providers that conduct regular security audits and make those reports available to their customers.
• Access Controls: Comprehensive access control options, including two-factor authentication (2FA) and role-based access, are essential.

Jurisdictional Considerations

• Non-PRISM Countries: Opt for providers based in countries not part of the PRISM surveillance program, enhancing privacy protection.
• Data Sovereignty: Understand where your data is physically stored and the legal implications of that jurisdiction.

Things to Avoid

• Free or Extremely Cheap Offers: These can be indicative of compromised security measures and a lack of investment in infrastructure.
• Providers with a History of Outages and Breaches: Research your provider's track record for reliability and security incidents.
• PRISM owned providers: Avoid providers that are incorporated and registered in PRISM based countries.

Billing Considerations

When comparing the cost implications of using usage-based services like Amazon Web Services (AWS) to flat-rate VPS hosting options, it's important to consider how these billing models can impact your budget and operational costs.

AWS operates on a usage-based pricing model, offering flexibility and scalability but potentially leading to unpredictable and higher costs depending on usage patterns. This model can be beneficial for businesses with fluctuating needs, allowing them to scale resources up or down based on demand. However, for users with stable or predictable resource requirements, this flexibility comes at the cost of complexity in budgeting due to potential overages and the intricate pricing structures associated with different services and resources within AWS.

In contrast, flat-rate VPS hosting services offer a predictable monthly fee, simplifying budget management and financial planning. These services typically provide a fixed amount of resources (CPU, RAM, storage, bandwidth) for a set price, making it easier for users to predict their hosting expenses without worrying about variable costs based on resource consumption. This pricing model is particularly appealing for small to medium-sized projects, personal websites, or businesses with consistent resource needs, where budget predictability is a priority.

The choice between usage-based and flat-rate billing models should be based on your specific hosting needs, resource usage patterns, and budgetary constraints. For projects with dynamic resource demands, a usage-based model like AWS might offer the necessary flexibility, albeit with more complex budgeting requirements. For those with more predictable resource needs, a flat-rate VPS hosting service could provide cost savings and simpler financial planning.

Pay-As-You-Go

The "pay-as-you-go" billing model, while offering flexibility and scalability, carries inherent risks that can lead to unexpectedly high charges, particularly in scenarios of hacking, abuse, or misconfiguration.

Hacking and Security Breaches

In a pay-as-you-go model, if an account is compromised, hackers can quickly spin up resources for their own purposes, such as cryptocurrency mining or launching attacks, leading to significant financial liabilities for the account owner. Since billing is based on usage, the costs can escalate rapidly without immediate detection of the unauthorized activity.

Abuse and Misconfiguration

Similarly, abuse or misconfiguration can lead to spiraling costs. For instance, leaving unused resources running, improperly configured services, or deploying resources more extensive than necessary can all contribute to higher than anticipated charges. An application or service experiencing a sudden increase in usage, whether through legitimate traffic spikes or DDoS attacks, can also result in significant expenses.

Comparing with Flat-Rate Billing

On the other hand, flat-rate VPS hosting offers a predictable monthly or annual fee, providing a capped cost that can safeguard against the unpredictable expenses associated with the pay-as-you-go model. This predictability is particularly valuable for small to medium-sized projects or businesses with fixed budgets. However, it's important to note that while flat-rate plans offer cost predictability, they may lack the flexibility to scale resources dynamically in response to changing needs.

Mitigation Strategies

To mitigate the risks associated with the pay-as-you-go model:

• Monitoring and Alerts: Utilize cloud service monitoring tools to track resource usage and set up alerts for unusual activity.
• Security Best Practices: Implement robust security measures, including multi-factor authentication, regular audits, and strict access controls.
• Budget Management Tools: Use budget management features offered by cloud providers to set spending limits and avoid unexpected charges.

Examples of Providers Tapping Their Clients

The owner of a popular chat service found that their large providers Hetzner and Linode tapped their servers which was documented on Reddit and by other security researchers.

In this case there was no evidence of the servers being hacked, but a device that acted as a man in the middle was placed in order to decrypt the encrypted traffic in and out of the server.  This was done to servers at 2 different locations in Germany and by two different providers (Linode and Hetzner).

By choosing large providers and choosing countries that co-operate with the US based PRISM program in the EU and most parts of the world, you are putting your data and security at risk.

https://therecord.media/jabber-ru-alleged-government-wiretap-expired-tls-certificate

Note a similar occurrence happened in France as well:

https://www.wired.com/story/encrochat-phone-police-hacking-encryption-drugs/

In all cases the clients were not notified and it's unclear what, if any legal process was used and it also appears that providers simply comply with requests for data and access, even if it may be illegal or unwarranted.

This will be the goto to help solve e-mail delivery issues and talk about many practical issues that happen between developers, admins and scripts that send e-mail and do things that may not be acceptable or cause deliverability problems.

Sendmail Stuff

Edit /etc/mail/sendmail.mc

The problem is that if you send directly out from the server using the mail function, the Return-path of the e-mails will be username@thehostnameoftheserver.com.  Let's say the From Address is "hellothere@cooldomain.com", this is a problem becasue presumably you've signed the e-mail for "cooldomain.com" but some e-mail providers will also check the Return-path domain of "thehostnameoftheserver.com" and find it does not match and there is no signature for the hostname and this will cause e-mails to be rejected or sent to spam.

So if your server's hostname is abc123.com, then the return path is the user@abc123.com, and this obviously breaks more strict DKIM checks because e-mail servers like gmail.com will also expect a DKIM signature for the hostname of the server.

To make things worse, a lot of times the people/devs that generate these e-mails may not have access to the server or may not be allowed to change the hostname.

Fix it Using Sendmail's MASQUERADE

Below we can fix this by using MASUERADE_AS with realtechtalk.com, this will be the domain set in the Return-path and the MASQUERADE_DOMAIN is abc123.com, you can add more lines with extra domains you want to rewrite.

dnl MASQUERADE_AS(`realtechtalk.com')dnl
dnl FEATURE(masquerade_entire_domain)dnl
dnl MASQUERADE_DOMAIN(abc123)dnl
 

To apply the changes update sendmail.cf like this:

We have to generate the new config file

m4 sendmail.mc

Restart the sendmail service.

systemctl restart sendmail

OpenDKIM Settings

Be sure to set this option for /etc/opendkim.conf as this will sign for multiple domains/hosts in the e-mail:

MultipleSignatures Yes

Eg. if your return-path is abc123.com and your from address is hello.com, both need to be signed or they will fail DMARC and possibly be blocked or sent to spam.  This also assumes that you should correctly have the keys and configuration to sign all domains that may be a part of the e-mail (eg. our example should mean that you have keys for abc123.com and hello.com on your server that sends out). 

First we need a few extra packages:

apt update

apt install -y adduser libfontconfig1 musl sudo
wget https://dl.grafana.com/enterprise/release/grafana-enterprise_10.3.1_amd64.deb



Install / Enable Grafana

dpkg -i grafana-enterprise_10.3.1_amd64.deb

dpkg -i grafana-enterprise_10.3.1_amd64.deb
(Reading database ... 44309 files and directories currently installed.)
Preparing to unpack grafana-enterprise_10.3.1_amd64.deb ...
Unpacking grafana-enterprise (10.3.1) over (10.3.1) ...
Setting up grafana-enterprise (10.3.1) ...
Restarting grafana-server service... OK
root@Grafana-realtechtalk:~# apt purge grafana-enterprise
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be REMOVED:
  grafana-enterprise*
0 upgraded, 0 newly installed, 1 to remove and 94 not upgraded.
After this operation, 408 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 44309 files and directories currently installed.)
Removing grafana-enterprise (10.3.1) ...
Stopping and disabling grafana-server service...
Synchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable grafana-server
Removed /etc/systemd/system/multi-user.target.wants/grafana-server.service.
(Reading database ... 34213 files and directories currently installed.)
Purging configuration files for grafana-enterprise (10.3.1) ...
dpkg: warning: while removing grafana-enterprise, directory '/etc/grafana' not empty so not removed
dpkg: warning: while removing grafana-enterprise, directory '/usr/lib/systemd/system' not empty so not removed
root@Grafana-realtechtalk:~# dpkg -i grafana-enterprise_10.3.1_amd64.deb
Selecting previously unselected package grafana-enterprise.
(Reading database ... 34208 files and directories currently installed.)
Preparing to unpack grafana-enterprise_10.3.1_amd64.deb ...
Unpacking grafana-enterprise (10.3.1) ...
Setting up grafana-enterprise (10.3.1) ...
### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
 sudo /bin/systemctl daemon-reload
 sudo /bin/systemctl enable grafana-server
### You can start grafana-server by executing
 sudo /bin/systemctl start grafana-server

Enable Grafana

systemctl daemon-reload
systemctl enable grafana-server

### You can start grafana-server by executing
 sudo /bin/systemctl start grafana-server
Synchronizing state of grafana-server.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable grafana-server

Created symlink /etc/systemd/system/multi-user.target.wants/grafana-server.service → /lib/systemd/system/grafana-server.service.

 

Start Grafana

systemctl start grafana-server

Access Grafana

Browse to https://YourIP:3000 and login as admin/admin

Update Your Password

Welcome to the Grafana Dashboard!

References

https://grafana.com/grafana/download?pg=oss-graf&plcmt=hero-btn-1

This is for the situation that you're doing other things that may conflict or have your own custom rules and ufw keeps overriding iptables.

A lot of guides don't work, and even ufw reset does not work, because it still leaves the old ufw chains.

Here is what works to disable ufw completely

systemctl stop ufw


systemctl disable ufw

ufw disable


rm -f /etc/ufw/*


iptables -F

Now run this script to finish the job

This script will delete all of the old ufw chains.

for chain in `iptables -L|grep ufw|awk '{print $2}'`; do
iptables -X $chain
done

Do this in case:

Sometimes the default iptables policy for FORWARD, OUTPUT, INPUT will be drop

iptables --policy FORWARD ACCEPT

iptables --policy INPUT ACCEPT

iptables --policy OUTPUT ACCEPT

We can see all the old chains below before running the script and then after the script they are all deleted.

You are probably using some custom image or maybe this is some sort of container that didn't boot with systemd.

The solution is to use "reboot -f"

reboot -f

This will force the system/OS/VM/container to reboot.

Reference To Related Issue.

During a migration or other custom setup, it's possible you don't have a $HOME/Desktop and as a result the config file that controls this is set to just use your $HOME directory.

This is controlled in the file: $HOME/.config/user-dirs.dirs

As we can see below, Desktop, Downloads, Music and Pictures are set to $HOME.  We can edit any of these as we like.

# This file is written by xdg-user-dirs-update
# If you want to change or add directories, just edit the line you're
# interested in. All local changes will be retained on the next run.
# Format is XDG_xxx_DIR="$HOME/yyy", where yyy is a shell-escaped
# homedir-relative path, or XDG_xxx_DIR="/yyy", where /yyy is an
# absolute path. No other format is supported.
#
XDG_DESKTOP_DIR="$HOME/"
XDG_DOWNLOAD_DIR="$HOME/"
XDG_TEMPLATES_DIR="$HOME/Templates"
XDG_PUBLICSHARE_DIR="$HOME/Public"
XDG_DOCUMENTS_DIR="$HOME/Documents"
XDG_MUSIC_DIR="$HOME/"
XDG_PICTURES_DIR="$HOME/"
XDG_VIDEOS_DIR="$HOME/Videos"

Fixed:

# This file is written by xdg-user-dirs-update
# If you want to change or add directories, just edit the line you're
# interested in. All local changes will be retained on the next run.
# Format is XDG_xxx_DIR="$HOME/yyy", where yyy is a shell-escaped
# homedir-relative path, or XDG_xxx_DIR="/yyy", where /yyy is an
# absolute path. No other format is supported.
#
XDG_DESKTOP_DIR="$HOME/Desktop"
XDG_DOWNLOAD_DIR="$HOME/Downloads"
XDG_TEMPLATES_DIR="$HOME/Templates"
XDG_PUBLICSHARE_DIR="$HOME/Public"
XDG_DOCUMENTS_DIR="$HOME/Documents"
XDG_MUSIC_DIR="$HOME/Music"
XDG_PICTURES_DIR="$HOME/Pictures"
XDG_VIDEOS_DIR="$HOME/Videos"

Normally syntax highlighting is the default behavior. 

You can enable it manually by hitting :syntax on

But normally you'll have this error because of the missing package.

E319: Sorry, the command is not available in this version

Solution install the vim-gui-common package:

apt install vim-gui-common
 

After that you should find that the code is highlighted by default.

Happy coding!

Proxmox's documentation shows the following here.

Which mainly just says change /etc/hosts and /etc/hostname with your new hostname.

Here's what happens if you only do that:

If you just do the above, you will find you have an inaccessible original hostname that contains those VMs and you cannot move or stop them.

You need to go into /etc/pve/nodes

You'll see the following inside:

oldhostname

newhostname

1.) Move both of those somewhere else.

2.) Create a dir called "newhostname"

3.) Copy the contents of "oldhostname" to "newhostname"

It's Fixed!

find  /usr/share/zoneinfo/|sed s#"/usr/share/zoneinfo/"##g|grep "/"|grep -v posix|grep -v ^"Etc/"|grep -v ^right|grep -v ^"SystemV"

Africa/Addis_Ababa
Africa/Abidjan
Africa/Blantyre
Africa/Lusaka
Africa/Casablanca
Africa/Libreville
Africa/Asmara
Africa/Bujumbura
Africa/Dakar
Africa/Lagos
Africa/Malabo
Africa/Harare
Africa/Kigali
Africa/Mogadishu
Africa/Maseru
Africa/Tunis
Africa/Ouagadougou
Africa/Sao_Tome
Africa/Kinshasa
Africa/Asmera
Africa/Tripoli
Africa/Khartoum
Africa/Conakry
Africa/Nouakchott
Africa/Djibouti
Africa/Niamey
Africa/Bangui
Africa/Ndjamena
Africa/Kampala
Africa/Johannesburg
Africa/Algiers
Africa/Banjul
Africa/Lubumbashi
Africa/Monrovia
Africa/Timbuktu
Africa/Juba
Africa/Windhoek
Africa/Cairo
Africa/Porto-Novo
Africa/Lome
Africa/Douala
Africa/Bissau
Africa/Brazzaville
Africa/Nairobi
Africa/Bamako
Africa/Dar_es_Salaam
Africa/Ceuta
Africa/Freetown
Africa/El_Aaiun
Africa/Maputo
Africa/Mbabane
Africa/Accra
Africa/Luanda
Africa/Gaborone
Chile/Continental
Chile/EasterIsland
Brazil/DeNoronha
Brazil/West
Brazil/East
Brazil/Acre
Antarctica/Davis
Antarctica/Macquarie
Antarctica/Vostok
Antarctica/Casey
Antarctica/Syowa
Antarctica/South_Pole
Antarctica/Rothera
Antarctica/Palmer
Antarctica/McMurdo
Antarctica/Troll
Antarctica/Mawson
Antarctica/DumontDUrville
Canada/Eastern
Canada/Newfoundland
Canada/Saskatchewan
Canada/Central
Canada/Yukon
Canada/Atlantic
Canada/Pacific
Canada/Mountain
US/Arizona
US/Eastern
US/Central
US/Indiana-Starke
US/Hawaii
US/Alaska
US/Aleutian
US/Samoa
US/Pacific
US/Mountain
US/Michigan
US/East-Indiana
Mexico/BajaNorte
Mexico/General
Mexico/BajaSur
Indian/Mayotte
Indian/Mauritius
Indian/Reunion
Indian/Kerguelen
Indian/Cocos
Indian/Christmas
Indian/Antananarivo
Indian/Comoro
Indian/Mahe
Indian/Chagos
Indian/Maldives
Europe/Brussels
Europe/Tallinn
Europe/San_Marino
Europe/Isle_of_Man
Europe/Bucharest
Europe/Zagreb
Europe/Oslo
Europe/Astrakhan
Europe/Mariehamn
Europe/Gibraltar
Europe/Zaporozhye
Europe/Ulyanovsk
Europe/Volgograd
Europe/Vaduz
Europe/Vienna
Europe/Bratislava
Europe/Chisinau
Europe/Berlin
Europe/Guernsey
Europe/Kirov
Europe/Tirane
Europe/Sarajevo
Europe/Budapest
Europe/Warsaw
Europe/Dublin
Europe/Jersey
Europe/Minsk
Europe/Moscow
Europe/Monaco
Europe/Kiev
Europe/Ljubljana
Europe/Kyiv
Europe/Istanbul
Europe/Prague
Europe/Sofia
Europe/Rome
Europe/Zurich
Europe/Madrid
Europe/Uzhgorod
Europe/Malta
Europe/Luxembourg
Europe/Kaliningrad
Europe/Paris
Europe/Vilnius
Europe/Vatican
Europe/Copenhagen
Europe/Belgrade
Europe/Stockholm
Europe/Helsinki
Europe/Lisbon
Europe/Podgorica
Europe/Simferopol
Europe/Andorra
Europe/Samara
Europe/Athens
Europe/London
Europe/Amsterdam
Europe/Skopje
Europe/Saratov
Europe/Busingen
Europe/Belfast
Europe/Nicosia
Europe/Riga
Europe/Tiraspol
Atlantic/Reykjavik
Atlantic/Jan_Mayen
Atlantic/St_Helena
Atlantic/South_Georgia
Atlantic/Canary
Atlantic/Stanley
Atlantic/Cape_Verde
Atlantic/Bermuda
Atlantic/Faeroe
Atlantic/Madeira
Atlantic/Azores
Atlantic/Faroe
America/Merida
America/Maceio
America/Monterrey
America/Jujuy
America/Costa_Rica
America/Santiago
America/Panama
America/Dawson_Creek
America/Bahia
America/Los_Angeles
America/Danmarkshavn
America/La_Paz
America/Iqaluit
America/Cancun
America/Rankin_Inlet
America/Rosario
America/Grand_Turk
America/Bahia_Banderas
America/Kralendijk
America/Punta_Arenas
America/Menominee
America/Moncton
America/Cordoba
America/Port-au-Prince
America/Inuvik
America/Noronha
America/Argentina
America/Argentina/Ushuaia
America/Argentina/Jujuy
America/Argentina/Cordoba
America/Argentina/Tucuman
America/Argentina/Mendoza
America/Argentina/Rio_Gallegos
America/Argentina/San_Juan
America/Argentina/Buenos_Aires
America/Argentina/Salta
America/Argentina/La_Rioja
America/Argentina/Catamarca
America/Argentina/San_Luis
America/Argentina/ComodRivadavia
America/Miquelon
America/Coral_Harbour
America/Curacao
America/Mendoza
America/Metlakatla
America/Sitka
America/Eirunepe
America/Fortaleza
America/Tegucigalpa
America/Recife
America/Dominica
America/Rainy_River
America/Ciudad_Juarez
America/St_Barthelemy
America/Boa_Vista
America/Thunder_Bay
America/Buenos_Aires
America/Resolute
America/Campo_Grande
America/Atikokan
America/St_Vincent
America/Chihuahua
America/Godthab
America/Dawson
America/Aruba
America/Winnipeg
America/El_Salvador
America/Porto_Velho
America/Caracas
America/Managua
America/Havana
America/Montevideo
America/Jamaica
America/Blanc-Sablon
America/Guayaquil
America/Port_of_Spain
America/Regina
America/Fort_Nelson
America/Denver
America/St_Thomas
America/Cayenne
America/Araguaina
America/Swift_Current
America/Anchorage
America/Antigua
America/Fort_Wayne
America/Barbados
America/Belize
America/Matamoros
America/Creston
America/Santarem
America/Nipigon
America/Louisville
America/Santo_Domingo
America/Halifax
America/Martinique
America/Chicago
America/Hermosillo
America/Rio_Branco
America/Lower_Princes
America/Manaus
America/Knox_IN
America/Phoenix
America/North_Dakota
America/North_Dakota/New_Salem
America/North_Dakota/Center
America/North_Dakota/Beulah
America/Tijuana
America/Boise
America/Bogota
America/Lima
America/Montreal
America/Indianapolis
America/Yellowknife
America/Toronto
America/Catamarca
America/Goose_Bay
America/Nuuk
America/Thule
America/Asuncion
America/Kentucky
America/Kentucky/Monticello
America/Kentucky/Louisville
America/Guatemala
America/Porto_Acre
America/Edmonton
America/Nassau
America/Cayman
America/Glace_Bay
America/Indiana
America/Indiana/Winamac
America/Indiana/Vevay
America/Indiana/Vincennes
America/Indiana/Marengo
America/Indiana/Petersburg
America/Indiana/Indianapolis
America/Indiana/Knox
America/Indiana/Tell_City
America/Vancouver
America/Tortola
America/Grenada
America/Mazatlan
America/Cambridge_Bay
America/Mexico_City
America/Paramaribo
America/Guyana
America/St_Kitts
America/Pangnirtung
America/Montserrat
America/Atka
America/Ensenada
America/Detroit
America/Shiprock
America/Whitehorse
America/Nome
America/Ojinaga
America/Juneau
America/Belem
America/St_Johns
America/Anguilla
America/St_Lucia
America/Puerto_Rico
America/New_York
America/Marigot
America/Guadeloupe
America/Sao_Paulo
America/Adak
America/Scoresbysund
America/Virgin
America/Cuiaba
America/Yakutat
America/Santa_Isabel
Arctic/Longyearbyen
Pacific/Tarawa
Pacific/Guam
Pacific/Gambier
Pacific/Easter
Pacific/Galapagos
Pacific/Chatham
Pacific/Fakaofo
Pacific/Majuro
Pacific/Pitcairn
Pacific/Pohnpei
Pacific/Noumea
Pacific/Bougainville
Pacific/Tahiti
Pacific/Wake
Pacific/Kosrae
Pacific/Ponape
Pacific/Niue
Pacific/Enderbury
Pacific/Rarotonga
Pacific/Honolulu
Pacific/Kiritimati
Pacific/Nauru
Pacific/Funafuti
Pacific/Tongatapu
Pacific/Guadalcanal
Pacific/Samoa
Pacific/Kanton
Pacific/Palau
Pacific/Pago_Pago
Pacific/Kwajalein
Pacific/Apia
Pacific/Johnston
Pacific/Saipan
Pacific/Efate
Pacific/Truk
Pacific/Norfolk
Pacific/Marquesas
Pacific/Port_Moresby
Pacific/Fiji
Pacific/Wallis
Pacific/Midway
Pacific/Auckland
Pacific/Yap
Pacific/Chuuk
Asia/Ashkhabad
Asia/Manila
Asia/Amman
Asia/Thimbu
Asia/Vientiane
Asia/Barnaul
Asia/Sakhalin
Asia/Irkutsk
Asia/Baku
Asia/Ashgabat
Asia/Saigon
Asia/Kabul
Asia/Aqtobe
Asia/Kuala_Lumpur
Asia/Khandyga
Asia/Yekaterinburg
Asia/Kuching
Asia/Beirut
Asia/Tel_Aviv
Asia/Macao
Asia/Taipei
Asia/Riyadh
Asia/Karachi
Asia/Pontianak
Asia/Ust-Nera
Asia/Jerusalem
Asia/Chita
Asia/Rangoon
Asia/Kuwait
Asia/Tehran
Asia/Brunei
Asia/Harbin
Asia/Srednekolymsk
Asia/Ujung_Pandang
Asia/Thimphu
Asia/Bangkok
Asia/Makassar
Asia/Phnom_Penh
Asia/Chongqing
Asia/Jayapura
Asia/Atyrau
Asia/Dacca
Asia/Anadyr
Asia/Dubai
Asia/Jakarta
Asia/Istanbul
Asia/Aden
Asia/Choibalsan
Asia/Macau
Asia/Almaty
Asia/Samarkand
Asia/Kamchatka
Asia/Yangon
Asia/Seoul
Asia/Gaza
Asia/Dhaka
Asia/Magadan
Asia/Kathmandu
Asia/Calcutta
Asia/Pyongyang
Asia/Tokyo
Asia/Chungking
Asia/Muscat
Asia/Vladivostok
Asia/Ho_Chi_Minh
Asia/Shanghai
Asia/Tashkent
Asia/Omsk
Asia/Katmandu
Asia/Ulan_Bator
Asia/Colombo
Asia/Hovd
Asia/Hong_Kong
Asia/Kolkata
Asia/Baghdad
Asia/Dili
Asia/Qatar
Asia/Kashgar
Asia/Hebron
Asia/Yerevan
Asia/Tbilisi
Asia/Aqtau
Asia/Urumqi
Asia/Yakutsk
Asia/Bishkek
Asia/Qostanay
Asia/Ulaanbaatar
Asia/Dushanbe
Asia/Novosibirsk
Asia/Novokuznetsk
Asia/Oral
Asia/Krasnoyarsk
Asia/Singapore
Asia/Nicosia
Asia/Qyzylorda
Asia/Bahrain
Asia/Tomsk
Asia/Famagusta
Asia/Damascus
Australia/Canberra
Australia/Darwin
Australia/Lord_Howe
Australia/West
Australia/Perth
Australia/NSW
Australia/Broken_Hill
Australia/Melbourne
Australia/LHI
Australia/Hobart
Australia/Lindeman
Australia/Yancowinna
Australia/Adelaide
Australia/Tasmania
Australia/North
Australia/Victoria
Australia/Eucla
Australia/South
Australia/Brisbane
Australia/Currie
Australia/Sydney
Australia/ACT
Australia/Queensland