network packets

Graphing network packets

This notebook currently relies on HoloViews 1.9 or above. Run conda install -c ioam/label/dev holoviews to install it.

Preparing data

The data source comes from a publicly available network forensics repository: http://www.netresec.com/?page=PcapFiles . The selected file is https://download.netresec.com/pcap/maccdc-2012/maccdc2012_00000.pcap.gz .

tcpdump -qns 0 -r maccdc2012_00000.pcap | grep tcp > maccdc2012_00000.txt

For example, here is a snapshot of the resulting output:

09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.24.100.1038 > 192.168.202.68.8080: tcp 0
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.27.100.37877 > 192.168.204.45.41936: tcp 0
09:30:07.780000 IP 192.168.24.100.1038 > 192.168.202.68.8080: tcp 0
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380
09:30:07.780000 IP 192.168.202.68.8080 > 192.168.24.100.1038: tcp 1380

Given the directional nature of network traffic and the numerous ports per node, we will simplify the graph by treating traffic between nodes as undirected and ignorning the distinction between ports. The graph edges will have weights represented by the total number of bytes across both nodes in either direction.

python pcap_to_parquet.py maccdc2012_00000.txt

The resulting output will be two Parquet dataframes, maccdc2012_nodes.parq and maccdc2012_edges.parq .

Loading data

In [1]:
import holoviews as hv
import networkx as nx
import dask.dataframe as dd

from holoviews.operation.datashader import (
    datashade, dynspread, directly_connect_edges, bundle_graph, stack
)
from holoviews.element.graphs import layout_nodes
from datashader.layout import random_layout
from colorcet import fire

hv.extension('bokeh')

%opts RGB Graph Nodes [bgcolor='black' width=800 height=800 xaxis=None yaxis=None]