GRR Rapid Response: remote live forensics for incident response

It appears that the frontends are constantly updating metadata:last on aff:/. Is this necessary? Having so many threads trying to hammer the same subject/attribute is causing delays. I haven't been able to find the code to see exactly why it is doing that.

I’m trying to let clients add labels via files. This is part of the config for my frontends that are doing the client repacking:

Client Context:
  Platform:Darwin:
    Config.includes:
      - build.yaml
      - "/etc/%(Client.name).labels.yaml"

  Platform:Linux:
    Config.includes:
      - build.yaml
      - "/etc/%(Client.name).labels.yaml"

  Platform:Windows:
    Config.includes:
      - build.yaml
      - "%(Client.install_path)/%(Client.binary_name).labels.yaml"

But when installing the mac .pkg, the Config.includes is never picked up:

$ cat /usr/local/lib/grr/grr_3.2.4.3_amd64/grr.yaml                                                                                                                                                                                                                                                                      
Client.arch: amd64
Client.company_name: GRR Project
Client.description: '%(name) %(platform) %(arch)'
Client.foreman_check_frequency: 1800
Client.install_path: /usr/local/lib/%(Client.name)/%(ClientRepacker.output_basename)
Client.name: grr
Client.platform: darwin
Client.plist_filename: '%(Client.plist_label).plist'
Client.plist_label: '%(Client.plist_label_prefix).google.code.%(Client.name)'
Client.plist_label_prefix: com
Client.plist_path: /Library/LaunchDaemons/%(Client.plist_filename)
Client.poll_max: 600
Client.rekall_profile_cache_path: '%(Client.install_path)/rekall_profiles'
Config.includes:
- build.yaml
Config.writeback: /etc/%(Client.name).local.yaml
Logging.engines: stderr,file,syslog
Logging.path: /var/log
Logging.syslog_path: /var/run/syslog
Logging.verbose: false
Client.deploy_time: '2018-11-01 07:57:29'

The repacking is using the right contexts and is picking up Config.includes, but I have no idea where /tmp/tmpZFFwpY/grr.yaml is coming from. Maybe it comes from the pre-baked OSX template?

Repacking template: /usr/share/grr-server/grr-response-templates/templates/grr_3.2.4.3_amd64.xar.zip
DEBUG:2018-12-28 22:08:18,114 8 MainProcess 140222624716544 MainThread config_lib:682] Applying filter env for CLIENT_INSTALLER_FINGERPRINT.
Using context: [u'ClientBuilder Context', u'ClientBuilder Context', u'Arch:amd64', u'Platform:Darwin', u'Target:Darwin', u'Target:Darwin'] and labels: []
DEBUG:2018-12-28 22:08:18,116 8 MainProcess 140222624716544 MainThread config_lib:1160] Loading configuration from /tmp/tmpZFFwpY/grr.yaml
DEBUG:2018-12-28 22:08:18,117 8 MainProcess 140222624716544 MainThread config_lib:850] Configuration writeback is set to /tmp/tmpZFFwpY/grr.yaml
...
DEBUG:2018-12-28 22:08:18,126 8 MainProcess 140222624716544 MainThread build:321] Copying config option to client: Config.includes
...
INFO:2018-12-28 22:08:18,131 8 MainProcess 140222624716544 MainThread config_lib:501] Writing back configuration to file /tmp/tmpZFFwpY/grr.yaml

Testing on tracking head. I decided to try to request approval for access from my own account. I received a notification on my account that I had requested access, and was able to approve it.

approvers.yaml looks like

label: "example"
requester_must_be_authorized: True
num_approvers_required: 1
users:
  - user1
  - user2
  - jessica

Below is a trimmed copy of the error encountered during grr_config_updater initialize portion. Confirmed efilter was latest version with pip. File "/usr/local/lib/python2.7/dist-packages/rekall/entities/init.py", line 3, in from rekall.entities import query File "/usr/local/lib/python2.7/dist-packages/rekall/entities/query/init.py", line 3, in from rekall.entities.query import analyzer File "/usr/local/lib/python2.7/dist-packages/rekall/entities/query/analyzer.py", line 29, in from efilter import engine ImportError: cannot import name engine

FAILURE RUNNING: grr_config_updater initialize

Ty.

...t behavior if no size is provided.

This causes problems when writing parsers that use external modules that expect a file_object and they call .read().

Getting this error in the server logs when I try running an AnalyzeClientMemory flow while specifying the pslist plugin:

ERROR:2016-09-12 18:17:55,761 flow_runner:618] Flow aff4:/C.c6259a0da13eab27/flows/F:8E4F11AB raised Error running plugins: Client action u'RekallAction' not known.
Traceback (most recent call last):
  File "/home/grr_user/GRR_NEW/local/lib/python2.7/site-packages/grr/lib/flow_runner.py", line 603, in RunStateMethod
    direct_response=direct_response, request=request, responses=responses)
  File "/home/grr_user/GRR_NEW/local/lib/python2.7/site-packages/grr/lib/flow.py", line 303, in Decorated
    res = f(*args[:f.func_code.co_argcount])
  File "/home/grr_user/GRR_NEW/local/lib/python2.7/site-packages/grr/lib/flows/general/memory.py", line 271, in End
    raise flow.FlowError("Error running plugins: %s" % all_errors)
FlowError: Error running plugins: Client action u'RekallAction' not known
ERROR:2016-09-12 18:17:55,813 flow_runner:937] Error in flow aff4:/C.c6259a0da13eab27/flows/F:8E4F11AB (aff4:/C.c6259a0da13eab27). Trace: Traceback (most recent call last):
  File "/home/grr_user/GRR_NEW/local/lib/python2.7/site-packages/grr/lib/flow_runner.py", line 603, in RunStateMethod
    direct_response=direct_response, request=request, responses=responses)
  File "/home/grr_user/GRR_NEW/local/lib/python2.7/site-packages/grr/lib/flow.py", line 303, in Decorated
    res = f(*args[:f.func_code.co_argcount])
  File "/home/grr_user/GRR_NEW/local/lib/python2.7/site-packages/grr/lib/flows/general/memory.py", line 271, in End
    raise flow.FlowError("Error running plugins: %s" % all_errors)
FlowError: Error running plugins: Client action u'RekallAction' not known

When I try to check server load for windows greater than 1hr the page never seems to finish loading. Watching the slow queries log on our SQL server seems to indicate it gets responses in a similar timeframe to the 1hr window that loads properly so it appears to be part of the processing to make the graphs.

Running UpdateClient from Ubuntu fails as the service restarts in the middle of the dpkg -i, killing all child processes leaving the service stopped and non-functional.

Jul 02 13:14:56 ubuntu grrd[13119]: (Reading database ... 141574 files and directories currently installed.)
Jul 02 13:14:56 ubuntu systemd[1]: Stopping grr linux amd64...
Jul 02 13:14:56 ubuntu systemd[1]: Stopped grr linux amd64.

I am seeing this error on 3.2.3.2 "Artifact LinuxUserProfiles missing from registry. You may need to sync the artifact repo by running make in the artifact directory."

I dont see a makefile in /grr/artifacts, however.

From aditya.kichu on January 08, 2014 08:33:38

What steps will reproduce the problem? 1. Built the linux client from source 2. Repacked it on the server and installed on linux client 3. Flows run on the linux client give runtime errors on new flows, works with existing flows. What is the expected output? What do you see instead? I have attached the output of the linux client build for reference. I want to confirm if the new client functionality has been included in the agent that is built. Also, I would like to know whether the agent has been properly built in the first place or not.

When I test the new flows that I created on this linux client, I see that the existing flows like Fingerprint File work properly without any errors, whereas the new flows that I added do not work properly even though they work perfectly in windows.

For example, I updated the Fingerprint files flow with fuzzy hashing, by adding new entries in the Fingerprint Tuple and updating my protobuf. However, when I run this flow it causes a Key error in the client action. Please see the error backtrace below.

Failed Fingerprint: message GrrStatus { backtrace : u'Traceback (most recent call last):\n File "/usr/local/grr_build/build/grr/out00-PYZ.pyz/grr.client.actions", line 127, in Execute\n File "/usr/local/grr_build/build/grr/out00-PYZ.pyz/grr.client.client_actions.file_fingerprint", line 47, in Run\nKeyError: 3\n' cpu_time_used : message CpuSeconds { system_cpu_time : 0.0 user_cpu_time : 0.0 } error_message : u'KeyError(3,): 3' network_bytes_sent : 384 status : GENERIC_ERROR }

The client side code for the Fingerprint File is the same as the original code except that there is one more hasher in the code that I have. This hasher is not recognized and causes the Key Error.

Another new flow that I created also does not run properly. How do I check whether the protobuf used in the client is the latest?

I am using GRR source code version 2.8.1.0 on Ubuntu 12.04 LTS.

It would be great if someone could help me in identifying the problem.

Thanks,

Aditya

Attachment: linux_client_build.txt

Original issue: http://code.google.com/p/grr/issues/detail?id=91

Dear users,

we have some real performance Issues with GRR at the moment. To give you a better understanding we use GRR Version 3.2.2.0 with MySQL and roughly 4000 Clients.

The GRR landscape is distributed across multiple servers (Ubuntu 16.04):

2x HTTP Frontend-server behind NGINX reverse Proxy
2x Worker Server (With 2 GRR-Worker processes)
1x UI-Server
1x MySQL-Datastore (with 2 GRR-Worker processes

At the moment we're not able to schedule any hunt on the system.

The logs show the following errors: "mysql_advanced_data_store: Operational Error: 1205 Lock wait timeout exceeded. Try restarting transaction. This may be due to an incorrect mysql "max_allowed_packet" setting (try increasing it).

We've tired values up to 2048MB, but this did not help. Another thing is that on the grr admin server we have a lot of socket in Status "Close_Wait".

I will add the current GRR-Settings and Mysql-Settings that we're using:

GRR:

Threadpool.size = 50
Worker.queue_shards = 8
Mysql.conn_pool_max = 50
Mysql.conn_pool_min = 10
Mysql.max_connect_wait = 0
Mysql.max_query_size = 8388608
Mysql.max_retries = 10
Mysql.max_values_per_query = 10000

Mysql: max_allowed_packet = 512M

Does anybody in this group have the same problems with the current grr ersion installed from the deb package? Or could you please give us some hints that may help gettimg grr up and running again?

Thanks for your help, Cheers Sven

Hello,

With Ubuntu 18.04LTS reaching end of life soon, and people moving to either 20.04 (python 3.8) or 22.04 (python 3.10), do you have any plans/schedules to release new .deb for these distributions?

Many thanks for that awesome tool, and merry xmas (a bit early)

-- certxlm

Bumps qs from 6.5.2 to 6.5.3.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps decode-uri-component from 0.2.0 to 0.2.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps minimatch from 3.0.4 to 3.1.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Hi, can I run GRR on single port? I want to use Cloudflare Tunnel but it work just with domain name (so 433 port for https) and does not support other ports. For docker deployment I need to have 8000 and 8080 port so I'd need separate domain for admin or client connection but I don't see this option either.