01
Human interaction through foundation models
Demo development focused on using foundation models to interpret what a person asks and adapt the robot’s task in response.
Technology
A field colleague, built as robotics software
GalacNo is developing software to connect existing site systems with robots, sensors and new equipment. The intended workflow starts with a request and ends with evidence a person can review.
Demo development
People, robots and a shared digital twin
We have been building demos of human interaction through foundation models, robot mapping and navigation in environments shared with people. Alongside these, our work includes a system to coordinate robots and humans, a digital twin of the activity, and an orchestration layer within that twin.
02
Mapping the environment
A robot mapping its surroundings to build a spatial view of the environment.
03
Navigating around people
A robot navigating environments shared with people, responding to movement and changes around it.
04
Coordinating robots and humans
A system coordinating work between robots and people in the same environment.
05
A digital twin of the activity
A digital representation of what is happening in the physical environment, bringing robot and human activity into a shared view.
06
Orchestration within the digital twin
An orchestration layer within the digital twin, connecting that shared view to the coordination of robots, people and tasks.
The workflow
Five functions, one supervised job
These functions describe the product being developed. Integration and behaviour must be validated for each site and job.
- 01
Understand the site
Bring together the work request, asset records, maps and available sensor data. Identify missing information before planning.
- 02
Plan the job
Propose the route, equipment and measurements, within the site’s access rules and operating limits.
- 03
Request approval
Ask the authorised site operator to review the scope, equipment and limits before work starts. Changes outside that scope need a new decision.
- 04
Coordinate equipment
Connect the approved job to the selected robots, sensors and site systems. Pause and hand back to an operator when the job cannot continue within its limits.
- 05
Record the result
Link captured data, timestamps, equipment status, approvals and exceptions to the original request for operator review.
People retain control
Approval, handback and evidence
Before work
An authorised operator approves
The site nominates the person authorised to approve the job. They review its scope, route, equipment, operating limits and stop conditions before execution.
During work
Pause when the plan no longer applies
The intended handback conditions include an obstacle, equipment fault, lost connection, insufficient sensor data or a request outside the approved scope. The configured stop or recovery procedure is agreed with the site; the operator decides whether to revise, resume or end the job.
After work
Keep a record that can be reviewed
The intended record includes the original request, approved plan, approver and time, equipment and software versions, captured data, interventions and completion status. The operator accepts the result or requests more work.
Technical detail
The architecture behind the workflow
“An OS for the physical world” is a supporting analogy: a common software layer connecting systems and coordinating work. The product is robotics software, and the scope of each integration matters.
Site understanding and shared maps
Sensor inputs and asset records contribute to a shared site model. The research architecture calls these components Hive-Mind and Sentient Node. Data freshness and missing coverage need to be visible to the planner and operator.
Job planning and equipment coordination
The orchestration layer, previously described as Fulcrum Fabric, connects a job plan to equipment adapters. Commands need to respect the approved scope, device capabilities and operating limits. Manipulation is a separate capability to validate where a job requires it.
Operator interface and monitoring
The interface presents requests, proposed plans, approvals and exceptions. The earlier Digital Teammate and Omni-Sense concepts inform interaction and monitoring; they do not establish a safety certification or replace site procedures.
Records, simulation and software changes
Simulation can help evaluate a plan before a controlled test. Recorded events should support review and replay. Software changes need version records, validation and an agreed recovery process before use at a site.