Troubleshoot Installation Issues

If you run into issues with your NetWitness IoT installation, refer to the following information to complete your installation and successfully set up NetWitness IoT.

Here are the issues and resolutions:

Issue 1: conntrack, forkstat not enabled

View log messages to check during agent startup

open /proc/sys/net/netfilter/nf_conntrack_timestamp: no such file or directory

procConnector =

procEvents =

netFilterNetlink =

nfConntrack = m

nfConntrackIPv4 =

nfConntrackIPv6 =

nfConntrackEvents = y

Written pid(1410523) to /var/run/rsa-rism-agent.pid

Hostname = gateway-test-machine

MachineID = 01f84b37c7eb9b8150988036a9890a476749e567ec78b7b575b9ded350a3dddc

Kernel version = 5.4.24-2.1.0+gbabac008e5cf #1 SMP PREEMPT Thu Dec 17 07:02:31 UTC 2020

Inside container = true

IP address = X.X.X.X

IP addresses = [X.X.X.X]

Local networks = [{X.X.X.X}]

nf_conntrack_acct =

nf_conntrack_timestamp =

ErrorLog: Unable to start forkstat() process monitoring: unable to create procnotify socket: Protocol not supported

ErrorLog: Unable to start conntrack() network monitoring: NETLINK error: unable to start conntrack: socket: protocol not supported

Resolution

Enable conntrack, forkstat modules and ensure nf_conntrack module and nf_conntrack_timestamp flags are set to true.

How to check if conntrack_timestamp is enabled/available in the module?

modinfo nf_conntrack

ls /proc/sys/net/netfilter/

Issue 2: Firewall blocks out-going traffic

If the network blocks outgoing traffic to AWS then the following error appears on the agent and Gateway will not be listed in NWIoT Cloud console.

Log messages to check in agent

AWS: Start authenticating

AWS: authenticate with Password

Upload data every 60 seconds

Upload all data every 60 seconds

Retrieve time from server

Get time from http://aws.amazon.com__;!!LpKI!zlJ4i- t0FyVdUvSvR4n9KZ1eukXUSFt4cdeCplh5x4BSy2aHtotGJy7-dTG8VOBE$ [aws[.]amazon[.]com]

Unable to get server time offset: Get "http://aws.amazon.com__;!!LpKI!zlJ4i- t0FyVdUvSvR4n9KZ1eukXUSFt4cdeCplh5x4BSy2aHtotGJy7-dTG8VOBE$ [aws[.]amazon[.]com]": dial tcp: lookup aws.amazon.com on [::1]:53: server misbehaving

Processes(kt,th,pr): new = (0,0,19), live = (0,0,0), dead = (0,0,18)

AWSS3: authentication error: RequestError: send request failed caused by: Post "https://cognito-idp.us-east-1.amazonaws.com/__!!LpKI!zlJ4i- t0FyVdUvSvR4n9KZ1eukXUSFt4cdeCplh5x4BSy2aHtotGJy7-dSHE7HSq$ [cognito-idp[.]us-east- 1[.]amazonaws[.]com]": dial``tcp: lookup cognito-idp.us-east-1.amazonaws.com on [::1]:53: server misbehaving

Resolution

Open up the following URLs or IPS in your firewall and make the following changes:

For Authentication to access our cloud analytics service:

URL     : 
           cognito-idp.us-east-1.amazonaws.com 	
Protocol: 
            HTTPS

For Time Sync:

 URL: 
         aws.amazon.com 
 Protocol: 
             http

For S3 (To upload metadata to cloud analytics service):

 Following are the list of IP ranges:

         54.231.0.0/16
         52.216.0.0/15
         3.5.0.0/19
         44.192.134.240/28
         44.192.140.64/28

 Protocol: 
             HTTPS

For cloudFront (Cloud Service access):

 Following are the list of IP ranges:

         3.231.2.0/25
         3.234.232.224/27
         3.236.169.192/26
         3.236.48.0/23
         34.195.252.0/24
        34.226.14.0/24

 Protocol: 
             HTTPS

NOTE:

Commands used to fetch S3 and CloudFront details:

jq -r '[.prefixes[] | select(.region=="us-east-1" and .service=="S3").ip_prefix] | .[]' < ip-ranges.json

jq -r '[.prefixes[] | select(.region=="us-east-1" and .service=="CLOUDFRONT").ip_prefix] | .[]' < ip-ranges.json

NOTE: ip-ranges.json is available at https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html

Issue 3: NTP/Time-sync issue with the Host

Log messages to check in the IoT agent

Get time from http://aws.amazon.com

Server time offset is 19787.481 seconds

upload.process: Unable to upload from Memory - RequestTimeTooSkewed: The difference between the request time and the current time is too large.

status code: 403, request id: V335V01AYQ3KNTYV, host id: zClioa8xksTUWSM6FQFu/SO0pQ7r6x69m098PAN8P0NxkQox2nER3xwdSaeGxe11tJr3NVa1V0k=

Resolution

Time on agent is out of sync. Sync time with any ntp server and re-run the agent.

Issue 4: How to capture docker agent logs and share with the support team for debugging?

Resolution

Redirect the Agent logs to nwiot-agent.log file

docker logs -f rism-agent >& nwiot-agent.log

Issue 5: Agent is listed as Gateway in cloud but no alerts are generated?

Resolution

If you run into this issue, contact NetWitness Customer Support.