Introducing The Lab

Introduction​

Almost a year ago, we quietly released an experimental version of The Lab with the goal of providing a public facing way to visualize the data we collect from the Xatu project. We didn't really announce it anywhere, and were using it as a playground to experiment with new ideas. While it was a hit, it was limited in functionality and labor intensive to add new features to. As a result it has been sitting dormant for the last few months.

The Lab​

We set out to build a new version of The Lab with a single north star: Make The Lab the community's first stop for verifying network performance. While we aren't quite there yet, we are aiming to rapidly iterate and improve the platform to get it to that point as soon as possible. Today's release aims to be at-parity with the previous experimental version of The Lab.

Since our team formed during The Merge, we've built numerous dashboards and tools to help us understand Ethereum network internals. This data requires deep context to interpret that our team already possesses. The Lab bridges this gap, allowing us to provide curated views of the data that are easy to understand.

Additionally, all the raw data that you see in The Lab will be directly available to the community. Ethereum is built on the principles of openness and transparency, and we strongly believe in the power of open data. We are committed to continuing to build on these principles in the form of the Xatu data project. For more info check out the New Clickhouse Tables section below.

Note: The Lab aims to provide a different perspective on the network that is complementary to existing explorers or monitoring tools that the community already uses, like Dora, Beaconcha.in, Etherscan, etc.

A fresh coat of paint​

We've completely redesigned The Lab with the polish and performance of a modern web application. This includes a new design system, 3 new themes (Light, Dark, and Star) and a refreshed approach to the user experience.

The Ethereum Live section within The Lab

Upcoming features​

We'll be adding a handful of new features to The Lab in the coming weeks. Some of these features are already in development but just need a little more time in the wok. Here's a sneak peek of what's coming:

• New pages in the Consensus section:
  • A new Node page which provides a detailed breakdown of the counts and versions of all the nodes on the network.
  • A new Validator page which provides a detailed view of a validator's performance over time, like when their attestation was seen by the Xatu nodes.
• A new Data Availability section including analysis of:
  • A "Custody" deepdive which explores the data availability of our peers on the network (in collaboration with the EF P2P Team)
• A new Execution Layer section including analysis of:
  • State size
  • OPCODE and precompile usage
  • Various state explorations, like size growth and hypothetical stateless explorations (in collaboration with the EF Stateless Consensus Team)
  • Client execution time - how long it takes for all Execution Layer clients to execute a block
• A new Fork Viewer section which follows in the footsteps of Forky by providing a visual representation of the head of the chain of our peers.

A transformational backend​

We went back to the drawing board on a lot of the fundamental tooling in our existing data collection pipeline to help us achieve our goals. We ended up creating 8 new projects. Some of these projects are useful outside of the Ethereum ecosystem, and we're open sourcing them to the community.

These projects chain together to significantly simplify the process of building new features:

1. A new model is added to xatu-cbt
2. clickhouse-proto-gen generates the Protobuf schemas for the new model.
3. cbt-api generates the new OpenAPI spec, and the server implementation to route queries to Clickhouse.
4. lab uses the new OpenAPI spec to generate typescript types for interacting with the new model within the Lab frontend.

The entire pipeline runs automatically through code generation, requiring no manual configuration. 🚀

The power of CBT ​

The largest breakthrough in this new architecture is CBT . CBT lets us easily transform and compose data into interconnected tables, aggregating and consolidating it into new formats that build upon each other. The Lab is the most direct beneficiary of these new transformation tables, but the tables are also useful for other projects, both within ethPandaOps and the community.

This abstraction simplifies complexity while compounding the value of our data.

CBT has a web interface which provides a visual representation of the data transformation pipeline in real time. For example, the fct_block table is built on top of other CBT tables in the form of int_block_canonical and fct_block_head. We'll be making a blog post about the inner workings of CBT soon.

CBT Mainnet showing the dependencies of the fct_block table

New Clickhouse tables​

We'll be publishing over 40 new transformation tables to the Xatu Dataset in the coming weeks. These tables directly power The Lab, meaning everything you see in The Lab will be directly available to the community via the Xatu Dataset . These tables are nearly always faster to query than the existing raw tables that we already publish, so if you want to build something new on top of the data we publish, these tables will be a great place to start.

Wrapping up​

We're extremely excited to be launching The Lab and we hope you enjoy using it. We'd love to hear your feedback or suggestions, please reach out to us on Twitter or join the Xatu Telegram Group.