1. Proposed title of this feature request
Add runbook_url to alerts in the OCP UI

2. What is the nature and description of the request?
If an alert includes a runbook_url label, then it should appear in the UI for the alert as a link.

3. Why does the customer need this? (List the business requirements here)
Customer can easily reach the alert runbook and be able to address their issues.

4. List any affected packages or components.

Feature Overview
Provide CSI drivers to replace all the intree cloud provider drivers we currently have. These drivers will probably be released as tech preview versions first before being promoted to GA.

Goals

• Framework for rapid creation of CSI drivers for our cloud providers
• CSI driver for AWS EBS
• CSI driver for AWS EFS
• CSI driver for GCP
• CSI driver for Azure
• CSI driver for VMware vSphere
• CSI Driver for Azure Stack
• CSI Driver for Alicloud
• CSI Driver for IBM Cloud

Requirements

Out of Scope

This work will only cover the drivers themselves, it will not include

• enhancements to the CSI API framework
• the migration to said drivers from the the intree drivers
• work for non-cloud provider storage drivers (FC-SAN, iSCSI) being converted to CSI drivers

Background, and strategic fit
In a future Kubernetes release (currently 1.21) intree cloud provider drivers will be deprecated and replaced with CSI equivalents, we need the drivers created so that we continue to support the ecosystems in an appropriate way.

Assumptions

• Storage SIG won't move out the changeover to a later Kubernetes release

Customer Considerations
Customers will need to be able to use the storage they want.

Documentation Considerations

• Target audience: cluster admins
• Updated content: update storage docs to show how to use these drivers (also better expose the capabilities)

We need to continue to maintain specific areas within storage, this is to capture that effort and track it across releases.

Goals

• To allow OCP users and cluster admins to detect problems early and with as little interaction with Red Hat as possible.
• When Red Hat is involved, make sure we have all the information we need from the customer, i.e. in metrics / telemetry / must-gather.
• Reduce storage test flakiness so we can spot real bugs in our CI.

Requirements

Out of Scope

n/a

Background, and strategic fit
With the expected scale of our customer base, we want to keep load of customer tickets / BZs low

Assumptions

Customer Considerations

Documentation Considerations

• Target audience: internal
• Updated content: none at this time.

Notes

In progress:

• CSI certification flakes a lot. We should fix it before we start testing migration.
  • In progress (API server restarts...) https://bugzilla.redhat.com/show_bug.cgi?id=1865857 

• Get local-storage-operator and AWS EBS CSI driver operator logs in must-gather (OLM-managed operators are not included there)
  • In progress for LSO (must-gather script being included in image) https://bugzilla.redhat.com/show_bug.cgi?id=1756096

• CI flakes:
  • Configurable timeouts for e2e tests
    • Azure is slow and times out often
    • Cinder times out formatting volumes
    • AWS resize test times out

High prio:

• Env. check tool for VMware - users often mis-configure permissions there and blame OpenShift. If we had a tool they could run, it might report better errors.
  • Should it be part of the installer?
  • Spike exists

• Add / use cloud API call metrics

• • Helps customers to understand why things are slow
  • Helps build cop to understand a flake
    • With a post-install step that filters data from Prometheus that’s still running in the CI job.
  • Ideas:
    • Cloud is throttling X% of API calls longer than Y seconds
    • Attach / detach / provisioning / deletion / mount / unmount / resize takes longer than X seconds?
  • Capture metrics of operations that are stuck and won’t finish.
    • Sweep operation map from executioner???
    • Report operation metric into the highest bucket after the bucket threshold (i.e. if 10minutes is the last bucket, report an operation into this bucket after 10 minutes and don’t wait for its completion)?
    • Ask the monitoring team?
  • Include in CSI drivers too.
    • With alerts too

• Report events for cloud issues
  • E.g. cloud API reports weird attach/provision error (e.g. due to outage)
• What volume plugins actually users use the most? https://issues.redhat.com/browse/STOR-324

Unsorted

• As the number of storage operators grows, it would be grafana board for storage operators
  • CSI driver metrics (from CSI sidecars + the driver itself  + its operator?)
  • CSI migration?

• Get aggregated logs in cluster
  • They're rotated too soon
  • No logs from dead / restarted pods
  • No tools to combine logs from multiple pods (e.g. 3 controller managers)

• What storage issues customers have? it was 22% of all issues.
  • Insufficient docs?
  • Probably garbage

• Document basic storage troubleshooting for our supports
  • What logs are useful when, what log level to use
  • This has been discussed during the GSS weekly team meeting; however, it would be beneficial to have this documented.

• Common vSphere errors, their debugging and fixing. 

• Document sig-storage flake handling - not all failed [sig-storage] tests are ours

Epic Goal

• Enable the migration from a storage intree driver to a CSI based driver with minimal impact to the end user, applications and cluster
• These migrations would include, but are not limited to:
  • CSI driver for AWS EBS
  • CSI driver for GCP
  • CSI driver for Azure (file and disk)
  • CSI driver for VMware vSphere

Why is this important?

• OpenShift needs to maintain it's ability to enable PVCs and PVs of the main storage types
• CSI Migration is getting close to GA, we need to have the feature fully tested and enabled in OpenShift
• Upstream intree drivers are being deprecated to make way for the CSI drivers prior to intree driver removal

Scenarios

1. User initiated move to from intree to CSI driver
2. Upgrade initiated move from intree to CSI driver
3. Upgrade from EUS to EUS

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Feature Overview

• As an infrastructure owner, I want a repeatable method to quickly deploy the initial OpenShift cluster.

• As an infrastructure owner, I want to install the first (management, hub, “cluster 0”) cluster to manage other (standalone, hub, spoke, hub of hubs) clusters.

Goals

• Enable customers and partners to successfully deploy a single “first” cluster in disconnected, on-premises settings

Requirements

4.11 MVP Requirements

• Customers and partners needs to be able to download the installer
• Enable customers and partners to deploy a single “first” cluster (cluster 0) using single node, compact, or highly available topologies in disconnected, on-premises settings
• Installer must support advanced network settings such as static IP assignments, VLANs and NIC bonding for on-premises metal use cases, as well as DHCP and PXE provisioning environments.
• Installer needs to support automation, including integration with third-party deployment tools, as well as user-driven deployments.
• In the MVP automation has higher priority than interactive, user-driven deployments.
• For bare metal deployments, we cannot assume that users will provide us the credentials to manage hosts via their BMCs.
• Installer should prioritize support for platforms None, baremetal, and VMware.
• The installer will focus on a single version of OpenShift, and a different build artifact will be produced for each different version.
• The installer must not depend on a connected registry; however, the installer can optionally use a previously mirrored registry within the disconnected environment.

Use Cases

• As a Telco partner engineer (Site Engineer, Specialist, Field Engineer), I want to deploy an OpenShift cluster in production with limited or no additional hardware and don’t intend to deploy more OpenShift clusters [Isolated edge experience].
• As a Enterprise infrastructure owner, I want to manage the lifecycle of multiple clusters in 1 or more sites by first installing the first  (management, hub, “cluster 0”) cluster to manage other (standalone, hub, spoke, hub of hubs) clusters [Cluster before your cluster].
• As a Partner, I want to package OpenShift for large scale and/or distributed topology with my own software and/or hardware solution.
• As a large enterprise customer or Service Provider, I want to install a “HyperShift Tugboat” OpenShift cluster in order to offer a hosted OpenShift control plane at scale to my consumers (DevOps Engineers, tenants) that allows for fleet-level provisioning for low CAPEX and OPEX, much like AKS or GKE [Hypershift].
• As a new, novice to intermediate user (Enterprise Admin/Consumer, Telco Partner integrator, RH Solution Architect), I want to quickly deploy a small OpenShift cluster for Poc/Demo/Research purposes.

Questions to answer…

Out of Scope

Out of scope use cases (that are part of the Kubeframe/factory project):

• As a Partner (OEMs, ISVs), I want to install and pre-configure OpenShift with my hardware/software in my disconnected factory, while allowing further (minimal) reconfiguration of a subset of capabilities later at a different site by different set of users (end customer) [Embedded OpenShift].
• As an Infrastructure Admin at an Enterprise customer with multiple remote sites, I want to pre-provision OpenShift centrally prior to shipping and activating the clusters in remote sites.

Background, and strategic fit

• This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

1. The user has only access to the target nodes that will form the cluster and will boot them with the image presented locally via a USB stick. This scenario is common in sites with restricted access such as government infra where only users with security clearance can interact with the installation, where software is allowed to enter in the premises (in a USB, DVD, SD card, etc.) but never allowed to come back out. Users can't enter supporting devices such as laptops or phones.
2. The user has access to the target nodes remotely to their BMCs (e.g. iDrac, iLo) and can map an image as virtual media from their computer. This scenario is common in data centers where the customer provides network access to the BMCs of the target nodes.
3. We cannot assume that we will have access to a computer to run an installer or installer helper software.

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

References

• Bootable Ephemeral Installer Image for Disconnected Cluster 0 Deployments

Feature Overview

Add GA support for deploying OpenShift to IBM Public Cloud

Goals

Complete the existing gaps to make OpenShift on IBM Cloud VPC (Next Gen2) General Available

Requirements

• OpenShift can be deployed into an existing network provided by the user (https://issues.redhat.com/browse/CORS-2263)
• OpenShift can be deployed with both Public and Private publish options (https://issues.redhat.com/browse/SPLAT-731)

Optional requirements

• OpenShift can be deployed using Mint mode and STS for cloud provider credentials (future release, tbd)
• OpenShift can be deployed in disconnected mode https://issues.redhat.com/browse/SPLAT-737)
• OpenShift on IBM Cloud supports User Provisioned Infrastructure (UPI) deployment method (future release, 4.14?)

Background and Goal

Currently in OpenShift we do not support distributing hotfix packages to cluster nodes. In time-sensitive situations, a RHEL hotfix package can be the quickest route to resolving an issue. 

Acceptance Criteria

1. Under guidance from Red Hat CEE, customers can deploy RHEL hotfix packages to MachineConfigPools.
2. Customers can easily remove the hotfix when the underlying RHCOS image incorporates the fix.

Why?

• Decouple control and data plane. 
  • Customers do not pay Red Hat more to run HyperShift control planes and supporting infrastructure than Standalone control planes and supporting infrastructure.
• Improve security
  • Shift credentials out of cluster that support the operation of core platform vs workload
• Improve cost
  • Allow a user to toggle what they don’t need.
  • Ensure a smooth path to scale to 0 workers and upgrade with 0 workers.

Assumption

• A customer will be able to associate a cluster as “Infrastructure only”
• E.g. one option: management cluster has role=master, and role=infra nodes only, control planes are packed on role=infra nodes
• OR the entire cluster is labeled infrastructure , and node roles are ignored.

• Anything that runs on a master node by default in Standalone that is present in HyperShift MUST be hosted and not run on a customer worker node.

Doc: https://docs.google.com/document/d/1sXCaRt3PE0iFmq7ei0Yb1svqzY9bygR5IprjgioRkjc/edit 

Feature Overview

We drive OpenShift cross-market customer success and new customer adoption with constant improvements and feature additions to the existing capabilities of our OpenShift Core Networking (SDN and Network Edge). This feature captures that natural progression of the product.

Goals

• Feature enhancements (performance, scale, configuration, UX, ...)
• Modernization (incorporation and productization of new technologies)

Requirements

• Core Networking Stability
• Core Networking Performance and Scale
• Core Neworking Extensibility (Multus CNIs)
• Core Networking UX (Observability)
• Core Networking Security and Compliance

In Scope

• Network Edge (ingress, DNS, LB)
• SDN (CNI plugins, openshift-sdn, OVN, network policy, egressIP, egress Router, ...)
• Networking Observability

Out of Scope

There are definitely grey areas, but in general:

• CNV
• Service Mesh
• CNF

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

This is a epic bucket for all activities surrounding the creation of declarative approach to release and maintain OLM catalogs.

tldr: three basic claims, the rest is explanation and one example

1. We cannot improve long term maintainability solely by fixing bugs.
2. Teams should be asked to produce designs for improving maintainability/debugability.
3. Specific maintenance items (or investigation of maintenance items), should be placed into planning as peer to PM requests and explicitly prioritized against them.

While bugs are an important metric, fixing bugs is different than investing in maintainability and debugability. Investing in fixing bugs will help alleviate immediate problems, but doesn't improve the ability to address future problems. You (may) get a code base with fewer bugs, but when you add a new feature, it will still be hard to debug problems and interactions. This pushes a code base towards stagnation where it gets harder and harder to add features.

One alternative is to ask teams to produce ideas for how they would improve future maintainability and debugability instead of focusing on immediate bugs. This would produce designs that make problem determination, bug resolution, and future feature additions faster over time.

I have a concrete example of one such outcome of focusing on bugs vs quality. We have resolved many bugs about communication failures with ingress by finding problems with point-to-point network communication. We have fixed the individual bugs, but have not improved the code for future debugging. In so doing, we chase many hard to diagnose problem across the stack. The alternative is to create a point-to-point network connectivity capability. this would immediately improve bug resolution and stability (detection) for kuryr, ovs, legacy sdn, network-edge, kube-apiserver, openshift-apiserver, authentication, and console. Bug fixing does not produce the same impact.

We need more investment in our future selves. Saying, "teams should reserve this" doesn't seem to be universally effective. Perhaps an approach that directly asks for designs and impacts and then follows up by placing the items directly in planning and prioritizing against PM feature requests would give teams the confidence to invest in these areas and give broad exposure to systemic problems.

Relevant links:

• Documentation:
  • Edge Diagnostics Scratchpad, our team's internal diagnostic guide.
  • Troubleshooting OCP networking issues - The complete guide, the SDN team's diagnostic guide.
  • Linux Performance, Brendan Gregg's guide to analyzing Linux performance issues.
  • RFC: A proper feedback loop on Alerts.
  • OpenShift Router Reload Technical Overview on Access.
  • Performance Scaling HAProxy with OpenShift on Access.
  • How to collect worker metrics to troubleshoot CPU load, memory pressure and interrupt issues and networking on worker nodes in OCP 4 on Access.
  • OpenShift Performance and Scale Knowledge Base on Mojo, results from OpenShift scalability testing.
  • Scalability and performance, OCP 4.5 documentation about the router's currently known scalability limits.
  • Scaling OpenShift Container Platform HAProxy Router, OCP 3.11 documentation about the manual performance configuration that was possible in OCP 3.
  • Timing web requests with cURL and Chrome from the Cloudflare blog.
  • tcpdump advanced filters, some useful tcpdump commands.
  • OpenShift SDN - Networking, OCP 3.11 documentation on the SDN (useful background reading).
  • Ingress Operator and Controller Status Conditions, design document for improved status condition reporting.
  • Observability tips for HAProxy, a slide deck by Willy Tarreau.
  • Interesting Traces - Out of Order versus Retransmissions, analysis using tshark.
  • The PCP Book: A Complete Documentation of Performance Co-Pilot, by Yogesh Babar.
  • Debugging kernel networking bug, brief guide to using SystemTap on RHCOS.
  • Troubleshooting throughput issues from the OCP 4.5 documentation.
  • Troubleshooting OpenShift Clusters and Workloads.
  • Red Hat Enterprise Linux Network Performance Tuning Guide (PDF).
  • openshift/enhancements#289 stability: point to point network check, a diagnostic built into the kube-apiserver operator.
• Diagnostic tools:
  • dropwatch to watch for packet drops.
  • ethtool to check NIC configuration.
  • iovisor/bcc: BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more to trace and diagnose various issues in the networking stack.
  • r-curler to gather timing information about HTTP/HTTPS connections.
  • route-monitor, to monitor routes for reachability.
  • hping(3), a programmable packet generator.
  • OpenTracing / Jaeger in OpenShift.
  • node-problem-detector, a possible integration point for new diagnostics.
  • Using SystemTap by Brendan Gregg.
  • DTrace SystemTap cheatsheet (PDF).
• Visualization and more sophisticated diagnostic tools:
  • eldadru/ksniff, kubectl plugin for tcpdump & Wireshark.
  • ironcladlou/ditm, Dan's "Dan in the Middle" tool.
  • Skydive, network diagnostic and visualization tool.
  • ali, a "load testing tool capable of performing real-time analysis" with visualization.
• Testing tools:
  • stress-ng, a general stress-loading tool (CPU, filesystem, network, ...).
  • mb, the networking benchmarking tool written and used by Jiri Mencak from our Perf+Scale team.
• Case studies:
  • BZ1763206 is an example of diagnosing DNS latency/timeouts.
  • BZ1829779 Investigation details the diagnosis of route latency.
  • BZ1845545 is an example of diagnosing misconfigured DNS for an external LB.
  • Debugging network stalls on Kubernetes, from the GitHub Blog, about diagnosing Kubernetes performance issues related to ksoftirqd.

Feature Overview

• This Section:* High-Level description of the feature ie: Executive Summary

• Note: A Feature is a capability or a well defined set of functionality that delivers business value. Features can include additions or changes to existing functionality. Features can easily span multiple teams, and multiple releases.

Goals

• This Section:* Provide high-level goal statement, providing user context and expected user outcome(s) for this feature

• …

Requirements

• This Section:* A list of specific needs or objectives that a Feature must deliver to satisfy the Feature.. Some requirements will be flagged as MVP. If an MVP gets shifted, the feature shifts. If a non MVP requirement slips, it does not shift the feature.

(Optional) Use Cases

This Section: 

• Main success scenarios - high-level user stories

• Alternate flow/scenarios - high-level user stories

• ...

Questions to answer…

• ...

Out of Scope

• …

Background, and strategic fit

This Section: What does the person writing code, testing, documenting need to know? What context can be provided to frame this feature.

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?

• Does this feature have doc impact?  

• New Content, Updates to existing content,  Release Note, or No Doc Impact

• If unsure and no Technical Writer is available, please contact Content Strategy.

• What concepts do customers need to understand to be successful in [action]?

• How do we expect customers will use the feature? For what purpose(s)?

• What reference material might a customer want/need to complete [action]?

• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.

• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Feature Overview

RHEL CoreOS should be updated to RHEL 9.2 sources to take advantage of newer features, hardware support, and performance improvements.

Requirements

• RHEL 9.x sources for RHCOS builds starting with OCP 4.13 and RHEL 9.2.

(Optional) Use Cases

• 9.2 Preview via Layering No longer necessary assuming we stay the course of going all in on 9.2

Assumptions

• ...

Customer Considerations

• ...

Documentation Considerations

Questions to be addressed:

• What educational or reference material (docs) is required to support this product feature? For users/admins? Other functions (security officers, etc)?
• Does this feature have doc impact?
• New Content, Updates to existing content, Release Note, or No Doc Impact
• If unsure and no Technical Writer is available, please contact Content Strategy.
• What concepts do customers need to understand to be successful in [action]?
• How do we expect customers will use the feature? For what purpose(s)?
• What reference material might a customer want/need to complete [action]?
• Is there source material that can be used as reference for the Technical Writer in writing the content? If yes, please link if available.
• What is the doc impact (New Content, Updates to existing content, or Release Note)?

Overview 

HyperShift came to life to serve multiple goals, some are main near-term, some are secondary that serve well long-term. 

Main Goals for hosted control planes (HyperShift)

• Optimize OpenShift for Cost/footprint/ which improves our competitive stance against the *KSes
• Establish separation of concerns which makes it more resilient for SRE to manage their workload clusters (be it security, configuration management, etc).
• Simplify and enhance multi-cluster management experience especially since multi-cluster is becoming an industry need nowadays. 

Secondary Goals

HyperShift opens up doors to penetrate the market. HyperShift enables true hybrid (CP and Workers decoupled, mixed IaaS, mixed Arch,...). An architecture that opens up more options to target new opportunities in the cloud space. For more details on this one check: Hosted Control Planes (aka HyperShift) Strategy [Live Document]

Hosted Control Planes (HyperShift) Map 

To bring hosted control planes to our customers, we need the means to ship it. Today MCE is how HyperShift shipped, and installed so that customers can use it. There are two main customers for hosted-control-planes: 

• Self-managed: In that case, Red Hat would provide hosted control planes as a service that is managed and SREed by the customer for their tenants (hence “self”-managed). In this management model, our external customers are the direct consumers of the multi-cluster control plane as a servie. Once MCE is installed, they can start to self-service dedicated control planes. 

• Managed: This is OpenShift as a managed service, today we only “manage” the CP, and share the responsibility for other system components, more info here. To reduce management costs incurred by service delivery organizations which translates to operating profit (by reducing variable costs per control-plane), as well as to improve user experience, lower platform overhead (allow customers to focus mostly on writing applications and not concern themselves with infrastructure artifacts), and improve the cluster provisioning experience. HyperShift is shipped via MCE, and delivered to Red Hat managed SREs (same consumption route). However, for managed services, additional tooling needs to be refactored to support the new provisioning path. Furthermore, unlike self-managed where customers are free to bring their own observability stack, Red Hat managed SREs need to observe the managed fleet to ensure compliance with SLOs/SLIs/…

If you have noticed, MCE is the delivery mechanism for both management models. The difference between managed and self-managed is the consumer persona. For self-managed, it's the customer SRE for managed its the RH SRE. 

High-level Requirements

For us to ship HyperShift in the product (as hosted control planes) in either management model, there is a necessary readiness checklist that we need to satisfy. Below are the high-level requirements needed before GA: 

• Hosted control planes fits well with our multi-cluster story (with MCE)
• Hosted control planes APIs are stable for consumption  
• Customers are not paying for control planes/infra components.  
• Hosted control planes has an HA and a DR story
• Hosted control planes is in parity with top-level add-on operators 
• Hosted control planes reports metrics on usage/adoption
• Hosted control planes is observable  
• HyperShift as a backend to managed services is fully unblocked.

Please also have a look at our What are we missing in Core HyperShift for GA Readiness? doc. 

Hosted control planes fits well with our multi-cluster story

Multi-cluster is becoming an industry need today not because this is where trend is going but because it’s the only viable path today to solve for many of our customer’s use-cases. Below is some reasoning why multi-cluster is a NEED:

As a result, multi-cluster management is a defining category in the market where Red Hat plays a key role. Today Red Hat solves for multi-cluster via RHACM and MCE. The goal is to simplify fleet management complexity by providing a single pane of glass to observe, secure, police, govern, configure a fleet. I.e., the operand is no longer one cluster but a set, a fleet of clusters. 

HyperShift logically centralized architecture, as well as native separation of concerns and superior cluster lifecyle management experience, makes it a great fit as the foundation of our multi-cluster management story. 

Thus the following stories are important for HyperShift: 

• When lifecycling OpenShift clusters (for any OpenShift form factor) on any of the supported providers from MCE/ACM/OCM/CLI as a Cluster Service Consumer  (RH managed SRE, or self-manage SRE/admin):
• I want to be able to use a consistent UI so I can manage and operate (observe, govern,...) a fleet of clusters.
• I want to specify HA constraints (e.g., deploy my clusters in different regions) while ensuring acceptable QoS (e.g., latency boundaries) to ensure/reduce any potential downtime for my workloads. 

• When operating OpenShift clusters (for any OpenShift form factor) on any of the supported provider from MCE/ACM/OCM/CLI as a Cluster Service Consumer  (RH managed SRE, or self-manage SRE/admin):
• I want to be able to backup any critical data so I am able to restore them in case of hosting service cluster (management cluster) failure. 

Refs:

• Agenda/Notes - HyperShift/ACM UX Sync
• https://coreos.slack.com/archives/C02TPGYJ6NN/p1657117542026339 
• AWS UI  and  Agent Provisioning UI  mocks

Hosted control planes APIs are stable for consumption.

HyperShift is the core engine that will be used to provide hosted control-planes for consumption in managed and self-managed. 

Main user story:  When life cycling clusters as a cluster service consumer via HyperShift core APIs, I want to use a stable/backward compatible API that is less susceptible to future changes so I can provide availability guarantees. 

Ref: What are we missing in Core HyperShift for GA Readiness?

Customers are not paying for control planes/infra components. 

Customers do not pay Red Hat more to run HyperShift control planes and supporting infrastructure than Standalone control planes and supporting infrastructure.

Assumptions: 

• A customer will be able to associate a cluster as “Infrastructure only”
• E.g. one option: management cluster has role=master, and role=infra nodes only, control planes are packed on role=infra nodes
• OR the entire cluster is labeled infrastructure , and node roles are ignored.

• Anything that runs on a master node by default in Standalone that is present in HyperShift MUST be hosted and not run on a customer worker node.

HyperShift - proposed cuts from data plane

HyperShift has an HA and a DR story

When operating OpenShift clusters (for any OpenShift form factor) from MCE/ACM/OCM/CLI as a Cluster Service Consumer  (RH managed SRE, or self-manage SRE/admin) I want to be able to migrate CPs from one hosting service cluster to another:

• as means for disaster recovery in the case of total failure
• so that scaling pressures on a management cluster can be mitigated or a management cluster can be decommissioned.

More information: 

• Hypershift Cluster Migration
• https://issues.redhat.com/browse/HOSTEDCP-327 

Hosted control planes reports metrics on usage/adoption

To understand usage patterns and inform our decision making for the product. We need to be able to measure adoption and assess usage.

See Hosted Control Planes (aka HyperShift) Strategy [Live Document]

Hosted control plane is observable  

Whether it's managed or self-managed, it’s pertinent to report health metrics to be able to create meaningful Service Level Objectives (SLOs), alert of failure to meet our availability guarantees. This is especially important for our managed services path. 

HyperShift is in parity with top-level add-on operators

https://issues.redhat.com/browse/OCPPLAN-8901 

Unblock HyperShift as a backend to managed services

HyperShift for managed services is a strategic company goal as it improves usability, feature, and cost competitiveness against other managed solutions, and because managed services/consumption-based cloud services is where we see the market growing (customers are looking to delegate platform overhead). 

We should make sure our SD milestones are unblocked by the core team. 

Note 

This feature reflects HyperShift core readiness to be consumed. When all related EPICs and stories in this EPIC are complete HyperShift can be considered ready to be consumed in GA form. This does not describe a date but rather the readiness of core HyperShift to be consumed in GA form NOT the GA itself.

- GA date for self-managed will be factoring in other inputs such as adoption, customer interest/commitment, and other factors. 
- GA dates for ROSA-HyperShift are on track, tracked in milestones M1-7 (have a look at https://issues.redhat.com/browse/OCPPLAN-5771) 

Feature Overview (aka. Goal Summary)  

The MCO should properly report its state in a way that's consistent and able to be understood by customers, troubleshooters, and maintainers alike. 

Some customer cases have revealed scenarios where the MCO state reporting is misleading and therefore could be unreliable to base decisions and automation on.

In addition to correcting some incorrect states, the MCO will be enhanced for a more granular view of update rollouts across machines.

Epic Goal

• Update OpenShift components that are owned by the Builds + Jenkins Team to use Kubernetes 1.25

Why is this important?

• Our components need to be updated to ensure that they are using the latest bug/CVE fixes, features, and that they are API compatible with other OpenShift components.

Acceptance Criteria

• Existing CI/CD tests must be passing

This is epic tracks "business as usual" requirements / enhancements / bug fixing of Insights Operator.

This epic contains all the Dynamic Plugins related stories for OCP release-4.11 

Epic Goal

• Track all the stories under a single epic

Acceptance Criteria

This epic contains all the Dynamic Plugins related stories for OCP release-4.12

Epic Goal

• Track all the stories under a single epic

Acceptance Criteria

This epic contains all the OLM related stories for OCP release-4.12

Epic Goal

• Track all the stories under a single epic

An epic we can duplicate for each release to ensure we have a place to catch things we ought to be doing regularly but can tend to fall by the wayside.

Epic Goal

• Enable OpenShift IPI Installer to deploy OCP to a shared VPC in GCP.
• The host project is where the VPC and subnets are defined. Those networks are shared to one or more service projects.
• Objects created by the installer are created in the service project where possible. Firewall rules may be the only exception.
• Documentation outlines the needed minimal IAM for both the host and service project.

Why is this important?

• Shared VPC's are a feature of GCP to enable granular separation of duties for organizations that centrally manage networking but delegate other functions and separation of billing. This is used more often in larger organizations where separate teams manage subsets of the cloud infrastructure. Enterprises that use this model would also like to create IPI clusters so that they can leverage the features of IPI. Currently organizations that use Shared VPC's must use UPI and implement the features of IPI themselves. This is repetative engineering of little value to the customer and an increased risk of drift from upstream IPI over time. As new features are built into IPI, organizations must become aware of those changes and implement them themselves instead of getting them "for free" during upgrades.

Scenarios

1. Deploy cluster(s) into service project(s) on network(s) shared from a host project.

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

1. Proposed title of this feature request
Basic authentication for Helm Chart repository in helmchartrepositories.helm.openshift.io CRD.

2. What is the nature and description of the request?
As of v4.6.9, the HelmChartRepository CRD only supports client TLS authentication through spec.connectionConfig.tlsClientConfig.

3. Why do you need this? (List the business requirements here)
Basic authentication is widely used by many chart repositories managers (Nexus OSS, Artifactory, etc.)
Helm CLI also supports them with the helm repo add command.
https://helm.sh/docs/helm/helm_repo_add/

4. How would you like to achieve this? (List the functional requirements here)
Probably by extending the CRD:

spec:
connectionConfig:
username: username
password:
secretName: secret-name

The secret namespace should be openshift-config to align with the tlsClientConfig behavior.

5. For each functional requirement listed in question 4, specify how Red Hat and the customer can test to confirm the requirement is successfully implemented.
Trying to pull helm charts from remote private chart repositories that has disabled anonymous access and offers basic authentication.
E.g.: https://github.com/sonatype/docker-nexus

Epic Goal

• Support manifest lists by image streams and the integrated registry. Clients should be able to pull/push manifests lists from/into the integrated registry. They also should be able to import images via `oc import-image` and them pull them from the internal registry.

Why is this important?

• Manifest lists are becoming more and more popular. Customers want to mirror manifest lists into the registry and be able to pull them by digest.

Scenarios

1. Manifest lists can be pushed into the integrated registry
2. Imported manifests list can be pulled from the integrated registry
3. Image triggers work with manifest lists

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• Existing functionality shouldn't change its behavior

Dependencies (internal and external)

1. ...

Previous Work (Optional)

1. https://github.com/openshift/enhancements/blob/master/enhancements/manifestlist/manifestlist-support.md

Open questions

1. Can we merge creation of images without having the pruner?

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

This is a follow up Epic to https://issues.redhat.com/browse/MCO-144, which aimed to get in-place upgrades for Hypershift. This epic aims to capture additional work to focus on using CoreOS/OCP layering into Hypershift, which has benefits such as:

 - removing or reducing the need for ignition

 - maintaining feature parity between self-driving and managed OCP models

 - adding additional functionality such as hotfixes

We plan to build Ironic Container Images using RHEL9 as base image in OCP 4.12

This is required because the ironic components have abandoned support for CentOS Stream 8 and Python 3.6/3.7 upstream during the most recent development cycle that will produce the stable Zed release, in favor of CentOS Stream 9 and Python 3.8/3.9

More info on RHEL8 to RHEL9 transition in OCP can be found at https://docs.google.com/document/d/1N8KyDY7KmgUYA9EOtDDQolebz0qi3nhT20IOn4D-xS4

Epic Goal

• We need the installer to accept a LB type from user and then we could set type of LB in the following object.
  oc get ingress.config.openshift.io/cluster -o yaml
  Then we can fetch info from this object and reconcile the operator to have the NLB changes reflected.

This is an API change and we will consider this as a feature request.

Why is this important?

https://issues.redhat.com/browse/NE-799 Please check this for more details

Scenarios

https://issues.redhat.com/browse/NE-799 Please check this for more details

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. installer
2. ingress operator

Previous Work (Optional):

 No

Open questions::

N/A

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Make sure that the CSI driver automatically updates oVirt credentials when they are updated in OpenShift.

We need tests for the ovirt-csi-driver and the cluster-api-provider-ovirt. These tests help us to

• minimize bugs,
• reproduce and fix them faster and
• pin down current behavior of the driver

Also, having dedicated tests on lower levels with a smaller scope (unit, integration, ...) has the following benefits:

• fast feedback cycle (local test execution)
• developer in-code documentation
• easier onboarding for new contributers
• lower resource consumption

Goal

Provide a form driven experience to allow cluster admins to manage the perspectives to meet the ACs below.

Problem:

We have heard the following requests from customers and developer advocates:

• Some admins do not want to provide access to the Developer Perspective from the console
• Some admins do not want to provide non-priv users access to the Admin Perspective from the console

Acceptance criteria:

1. Cluster administrator is able to "hide" the admin perspective for non-priv users
2. Cluster administrator is able to "hide" the developer perspective for all users
3. Be user that User Preferences for individual users behaves appropriately. If only one perspective is available, the perspective switcher is not needed.

Dependencies (External/Internal):

Design Artifacts:

Exploration:

• Miro board: https://miro.com/app/board/uXjVOoSAPbA=/
• Recording: https://drive.google.com/file/d/10JUmaTMdfMH1NceHJW_74Y2cUxqwqmD_/view

Note:

Problem:

Customers don't want their users to have access to some/all of the items which are available in the Developer Catalog.  The request is to change access for the cluster, not per user or persona.

Goal:

Provide a form driven experience to allow cluster admins easily disable the Developer Catalog, or one or more of the sub catalogs in the Developer Catalog.

Why is it important?

Multiple customer requests.

Acceptance criteria:

1. As a cluster admin, I can hide/disable access to the developer catalog for all users across all namespaces.
2. As a cluster admin, I can hide/disable access to a specific sub-catalog in the developer catalog for all users across all namespaces.
   1. Builder Images
   2. Templates
   3. Helm Charts
   4. Devfiles
   5. Operator Backed

Notes

We need to consider how this will work with subcatalogs which are installed by operators: VMs, Event Sources, Event Catalogs, Managed Services, Cloud based services

Dependencies (External/Internal):

Design Artifacts:

Exploration:

Note:

Epic Goal

• Facilitate the transition to for OLM and content to PSA enforcing the `restricted` security profile
• Use the label synch'er to enforce the required security profile
• Current content should work out-of-the-box as is
• Upgrades should not be blocked

Why is this important?

• PSA helps secure the cluster by enforcing certain security restrictions that the pod must meet to be scheduled
• 4.12 will enforce the `restricted` profile, which will affect the deployment of operators in `openshift-*` namespaces 

Scenarios

1. Admin installs operator in an `openshift-*`namespace that is not managed by the label syncher -> label should be applied
2. Admin installs operator in an `openshift-*` namespace that has a label asking the label syncher to not reconcile it -> nothing changes

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• Done only downstream
• Transition documentation written and reviewed

Dependencies (internal and external)

1. label syncher (still searching for the link)

Open questions::

1. Is this only for openshift-* namespaces?

Resources

• notes on label syncer and what's needed for OLM
• PSA Communication - explains all required changes to adapt workloads to PSA under OCP 4.12 (inc. SCC)

Stakeholders

• Daniel S...?

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

This epic tracks network tooling improvements for 4.12

New framework and process should be developed to make sharing network tools with devs, support and customers convenient. We are going to add some tools for ovn troubleshooting before ovn-k goes default, also some tools that we got from customer cases, and some more to help analyze and debug collected logs based on stable must-gather/sosreport format we get now thanks to 4.11 Epic.

Our estimation for this Epic is 1 engineer * 2 Sprints

WHY:
This epic is important to help improve the time it takes our customers and our team to understand an issue within the cluster.
A focus of this epic is to develop tools to quickly allow debugging of a problematic cluster. This is crucial for the engineering team to help us scale. We want to provide a tool to our customers to help lower the cognitive burden to get at a root cause of an issue.

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

• Come up with a consistent way to detect node down on OCP and hypershift. Current mechanism for OCP (probe port 9) does not work for hypershift, meaning, hypershift node down detection will be longer (~40 secs). We should aim to have a common mechanism for both. As well, we should consider alternatives to the probing port 9. Perhaps BFD, or other detection.
• Get clarification on node down detection times. Some customers have (apparently) asked for detection on the order of 100ms, recommendation is to use multiple Egress IPs, so this may not be a hard requirement. Need clarification from PM/Customers.

Why is this important?

• …

Scenarios

1. ...

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

This Epic is here to track the rebase we need to do when kube 1.25 is GA https://www.kubernetes.dev/resources/release/

Keeping this in mind can help us plan our time better. ATTOW GA is planned for August 23

https://docs.google.com/document/d/1h1XsEt1Iug-W9JRheQas7YRsUJ_NQ8ghEMVmOZ4X-0s/edit --> this is the link for rebase help

Changes made in METAL-1 open up opportunities to improve our handling of images by cleaning up redundant code that generates extra work for the user and extra load for the cluster.

Epic Goal

• To improve the reliability of disk cleaning before installation and to provide the user with sufficient warning regarding the consequences of the cleaning

Why is this important?

• Insufficient cleaning can lead to installation failure
• Insufficient warning can lead to complaints of unexpected data loss

Scenarios

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Epic Goal

• Increase success-rate of of our CI jobs
• Improve debugability / visibility or tests 

Why is this important?

• Failed presubmit jobs (required or optional) can make an already tested+approved PR to not get in
• Failed periodic jobs interfere our visibility around stability of features

Section 5 of PRD: https://docs.google.com/document/d/1fF-Ajdzc9EDDg687FzTrX577hvY9NdK0/edit#heading=h.gjdgxs

Testing and collaboration with NVIDIA: https://docs.google.com/spreadsheets/d/1LHY-Af-2kQHVwtW4aVdHnmwZLTiatiyf-ySffC8O5NM/edit#gid=0 

Deploying Nvidia Patches:  https://docs.google.com/document/d/1yR4lphjPKd6qZ9sGzZITl0wH1r4ykfMKPjUnlzvWji4/edit# 

This is the continuation of https://issues.redhat.com/browse/NHE-273 but now the focus is on the remainig flows

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

• Run OpenShift builds that do not execute as the "root" user on the host node.

Why is this important?

• OpenShift builds require an elevated set of capabilities to build a container image
• Builds currently run as root to maintain adequate performance
• Container workloads should run as non-root from the host's perspective. Containers running as root are a known security risk.
• Builds currently run as root and require a privileged container. See BUILD-225 for removing the privileged container requirement.

Scenarios

1. Run BuildConfigs in a multi-tenant environment
2. Run BuildConfigs in a heightened security environment/deployment

Acceptance Criteria

• Developers can opt into running builds in a cri-o user namespace by providing an environment variable with a specific value.
• When the correct environment variable is provided, builds run in a cri-o user namespace, and the build pod does not require the "privileged: true" security context.
• User namespace builds can pass basic test scenarios for the Docker and Source strategy build.
• Steps to run unprivileged builds are documented.

Dependencies (internal and external)

1. Buildah supports running inside a non-privileged container
2. CRI-O allows workloads to opt into running containers in user namespaces.

Previous Work (Optional):

1. BUILD-225 - remove privileged requirement for builds.

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

OCP/Telco Definition of Done
Epic Template descriptions and documentation.

<--- Cut-n-Paste the entire contents of this description into your new Epic --->

Epic Goal

• ...

Why is this important?

• …

Scenarios

1. ...

Acceptance Criteria

• CI - MUST be running successfully with tests automated
• Release Technical Enablement - Provide necessary release enablement details and documents.
• ...

Dependencies (internal and external)

1. ...

Previous Work (Optional):

1. …

Open questions::

1. …

Done Checklist

• CI - CI is running, tests are automated and merged.
• Release Enablement <link to Feature Enablement Presentation>
• DEV - Upstream code and tests merged: <link to meaningful PR or GitHub Issue>
• DEV - Upstream documentation merged: <link to meaningful PR or GitHub Issue>
• DEV - Downstream build attached to advisory: <link to errata>
• QE - Test plans in Polarion: <link or reference to Polarion>
• QE - Automated tests merged: <link or reference to automated tests>
• DOC - Downstream documentation merged: <link to meaningful PR>

Goal
Provide an indication that advanced features are used

Problem

Today, customers and RH don't have the information on the actual usage of advanced features.

Why is this important?

1. Better focus upsell efforts
2. Compliance information for customers that are not aware they are not using the right subscription

Prioritized Scenarios

In Scope
1. Add a boolean variable in our telemetry to mark if the customer is using advanced features (PV encryption, encryption with KMS, external mode). 

Not in Scope

Integrate with subscription watch - will be done by the subscription watch team with our help.

Customers

All

Customer Facing Story
As a compliance manager, I should be able to easily see if all my clusters are using the right amount of subscriptions

What does success look like?

A clear indication in subscription watch for ODF usage (either essential or advanced). 

We migrated most component as part of https://issues.redhat.com/browse/RHSTOR-2165
We now have a few components remaining roughly 15 to 20%. This epic tragets

This is a clone of issue OCPBUGS-4913. The following is the description of the original issue:
—
Description of problem:

Currently the Terraform code waits for 45 seconds, but anecdotal data suggest we should actually wait for 3 minutes in order to avoid "failures" due to occasional slow boots of a new VM in PowerVS.

Version-Release number of selected component (if applicable):

How reproducible:

often enough

Steps to Reproduce:

1. run IPI installer against PowerVS
2. look for "empty tuple" in the error message when it fails to reach `bootstrap-complete`
3.

Actual results:

Expected results:

VMs to always have IP address assigned by DHCP after a certain wait

Additional info:

The change has already been merged into master/4.13, but 4.12 also needs this for planned PowerVS IPI GA on the z-stream.

• https://github.com/openshift/installer/pull/6709

This is a clone of issue OCPBUGS-6663. The following is the description of the original issue:
—
Description of problem:

When running openshift-install agent create image, and the install-config.yaml does not contain platform baremetal settings (except for VIPs) warnings are still generated as below:
DEBUG         Loading Install Config...            
WARNING Platform.Baremetal.ClusterProvisioningIP: 172.22.0.3 is ignored 
DEBUG Platform.Baremetal.BootstrapProvisioningIP: 172.22.0.2 is ignored 
WARNING Platform.Baremetal.ExternalBridge: baremetal is ignored 
WARNING Platform.Baremetal.ExternalMACAddress: 52:54:00:12:e1:68 is ignored 
WARNING Platform.Baremetal.ProvisioningBridge: provisioning is ignored 
WARNING Platform.Baremetal.ProvisioningMACAddress: 52:54:00:82:91:8d is ignored 
WARNING Platform.Baremetal.ProvisioningNetworkCIDR: 172.22.0.0/24 is ignored 
WARNING Platform.Baremetal.ProvisioningDHCPRange: 172.22.0.10,172.22.0.254 is ignored 
WARNING Capabilities: %!!(MISSING)s(*types.Capabilities=<nil>) is ignored 

It looks like these fields are populated with values from libvirt as shown in .openshift_install_state.json:
            "platform": {
                "baremetal": {
                    "libvirtURI": "qemu:///system",
                    "clusterProvisioningIP": "172.22.0.3",
                    "bootstrapProvisioningIP": "172.22.0.2",
                    "externalBridge": "baremetal",
                    "externalMACAddress": "52:54:00:12:e1:68",
                    "provisioningNetwork": "Managed",
                    "provisioningBridge": "provisioning",
                    "provisioningMACAddress": "52:54:00:82:91:8d",
                    "provisioningNetworkInterface": "",
                    "provisioningNetworkCIDR": "172.22.0.0/24",
                    "provisioningDHCPRange": "172.22.0.10,172.22.0.254",
                    "hosts": null,
                    "apiVIPs": [
                        "10.1.101.7",
                        "2620:52:0:165::7"
                    ],
                    "ingressVIPs": [
                        "10.1.101.9",
                        "2620:52:0:165::9"
                    ]

The install-config.yaml used to generate this has the following snippet:
platform:
  baremetal:
    apiVIPs:
    - 10.1.101.7
    - 2620:52:0:165::7
    ingressVIPs:
    - 10.1.101.9
    - 2620:52:0:165::9
additionalTrustBundle: |

Version-Release number of selected component (if applicable):

4.12.0

How reproducible:

Happens every time

Steps to Reproduce:

1. Use install-config.yaml with no platform baremetal fields except for the VIPs
2. run openshift-install agent create image 

Actual results:

Warning messages are output

Expected results:

No warning messags

Additional info:

• https://github.com/openshift/installer/pull/6815

This is a clone of issue OCPBUGS-2479. The following is the description of the original issue:
—
Description of problem:

Right border radius is 0 for the pipeline visualization wrapper in dark mode but looks fine in light mode

Version-Release number of selected component (if applicable):

4.12

How reproducible:

Steps to Reproduce:

1. Switch the theme to dark mode
2. Create a pipeline and navigate to the Pipeline details page

Actual results:

Right border radius is 0, see the screenshots

Expected results:

Right border radius should be same as left border radius.

Additional info:

• https://github.com/openshift/console/pull/12565

Description of problem:

Using a daemonset that causes failures during draining as leases are not gracefully released and instead age out as pods are killed after potentially losing network access due to daemonset pods not being terminated. 

As pointed out in https://github.com/openshift/origin/pull/27394#discussion_r964002900 


This should be fixed when moving to a deployment and is also tracked here https://issues.redhat.com/browse/BUILD-495 

Version-Release number of selected component (if applicable):

How reproducible:

100

Steps to Reproduce:

1. 
2. 
3.

Actual results:

Expected results:

Additional info:

• https://github.com/openshift/openshift-controller-manager/pull/239

This is a clone of issue OCPBUGS-3744. The following is the description of the original issue:
—
Description of problem:

Egress router POD creation on Openshift 4.11 is failing with below error.
~~~
Nov 15 21:51:29 pltocpwn03 hyperkube[3237]: E1115 21:51:29.467436    3237 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"stage-wfe-proxy-ext-qrhjw_stage-wfe-proxy(c965a287-28aa-47b6-9e79-0cc0e209fcf2)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"stage-wfe-proxy-ext-qrhjw_stage-wfe-proxy(c965a287-28aa-47b6-9e79-0cc0e209fcf2)\\\": rpc error: code = Unknown desc = failed to create pod network sandbox k8s_stage-wfe-proxy-ext-qrhjw_stage-wfe-proxy_c965a287-28aa-47b6-9e79-0cc0e209fcf2_0(72bcf9e52b199061d6e651e84b0892efc142601b2442c2d00b92a1ba23208344): error adding pod stage-wfe-proxy_stage-wfe-proxy-ext-qrhjw to CNI network \\\"multus-cni-network\\\": plugin type=\\\"multus\\\" name=\\\"multus-cni-network\\\" failed (add): [stage-wfe-proxy/stage-wfe-proxy-ext-qrhjw/c965a287-28aa-47b6-9e79-0cc0e209fcf2:openshift-sdn]: error adding container to network \\\"openshift-sdn\\\": CNI request failed with status 400: 'could not open netns \\\"/var/run/netns/8c5ca402-3381-4935-baed-ea454161d669\\\": unknown FS magic on \\\"/var/run/netns/8c5ca402-3381-4935-baed-ea454161d669\\\": 1021994\\n'\"" pod="stage-wfe-proxy/stage-wfe-proxy-ext-qrhjw" podUID=c965a287-28aa-47b6-9e79-0cc0e209fcf2
~~~

I have checked SDN POD log from node where egress router POD is failing and I could see below error message.

~~~
2022-11-15T21:51:29.283002590Z W1115 21:51:29.282954  181720 pod.go:296] CNI_ADD stage-wfe-proxy/stage-wfe-proxy-ext-qrhjw failed: could not open netns "/var/run/netns/8c5ca402-3381-4935-baed-ea454161d669": unknown FS magic on "/var/run/netns/8c5ca402-3381-4935-baed-ea454161d669": 1021994
~~~

Crio is logging below event and looking at the log it seems the namespace has been created on node.

~~~
Nov 15 21:51:29 pltocpwn03 crio[3150]: time="2022-11-15 21:51:29.307184956Z" level=info msg="Got pod network &{Name:stage-wfe-proxy-ext-qrhjw Namespace:stage-wfe-proxy ID:72bcf9e52b199061d6e651e84b0892efc142601b2442c2d00b92a1ba23208344 UID:c965a287-28aa-47b6-9e79-0cc0e209fcf2 NetNS:/var/run/netns/8c5ca402-3381-4935-baed-ea454161d669 Networks:[] RuntimeConfig:map[multus-cni-network:{IP: MAC: PortMappings:[] Bandwidth:<nil> IpRanges:[]}] Aliases:map[]}"
~~~

Version-Release number of selected component (if applicable):

4.11.12

How reproducible:

Not Sure

Steps to Reproduce:

1.
2.
3.

Actual results:

Egress router POD is failing to create. Sample application could be created without any issue.

Expected results:

Egress router POD should get created

Additional info:

Egress router POD is created following below document and it does contain pod.network.openshift.io/assign-macvlan: "true" annotation.

https://docs.openshift.com/container-platform/4.11/networking/openshift_sdn/deploying-egress-router-layer3-redirection.html#nw-egress-router-pod_deploying-egress-router-layer3-redirection

• https://github.com/openshift/cluster-network-operator/pull/1628

This is a clone of issue OCPBUGS-4181. The following is the description of the original issue:
—
Description of problem:

After configuring a webhook receiver in alertmanager to send alerts to an external tool, a customer noticed that when receiving alerts they have as source "https:///<console-url>" (notice the 3 /).

Version-Release number of selected component (if applicable):

OCP 4.10

How reproducible:

Always

Steps to Reproduce:

1.
2.
3.

Actual results:

https:///<console-url>

Expected results:

https://<console-url>

Additional info:

After investigating I discovered that the problem might be in the CMO code:

→ oc get Alertmanager main -o yaml | grep externalUrl
  externalUrl: https:/console-openshift-console.apps.jakumar-2022-11-27-224014.devcluster.openshift.com/monitoring
→ oc get Prometheus k8s -o yaml | grep externalUrl
  externalUrl: https:/console-openshift-console.apps.jakumar-2022-11-27-224014.devcluster.openshift.com/monitoring

• https://github.com/openshift/cluster-monitoring-operator/pull/1840

Description of problem:

TestEditUnmanagedPodDisruptionBudget flakes in the console-operator e2e

Version-Release number of selected component (if applicable):

4.12

How reproducible:

Flake

Steps to Reproduce:
1. Check https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_console-operator/665/pull-ci-openshift-console-operator-master-e2e-aws-operator/1562005782164148224
2.
3.

Actual results:

Expected results:

Additional info:

There is a chance that the PDB instances is not present since prior to the Unmanaged* TCs the RemoveTest is running which is removing all the console resources (Pods, Services, PDBs, ...).

• https://github.com/openshift/console-operator/pull/675

This is a clone of issue OCPBUGS-3993. The following is the description of the original issue:
—
Description of problem:
On Openshift on Openstack CI, we are deploying an OCP cluster with an additional network on the workers in install-config.yaml for integration with Openstack Manila.

compute:
- name: worker
  platform:
    openstack:
      zones: []
      additionalNetworkIDs: ['0eeae16f-bbc7-4e49-90b2-d96419b7c30d']
  replicas: 3

As a result, the egressIP annotation includes two interfaces definition:

$ oc get node ostest-hp9ld-worker-0-gdp5k -o json | jq -r '.metadata.annotations["cloud.network.openshift.io/egress-ipconfig"]' | jq .                                 
[
  {
    "interface": "207beb76-5476-4a05-b412-d0cc53ab00a7",
    "ifaddr": {
      "ipv4": "10.46.44.64/26"
    },
    "capacity": {
      "ip": 8
    }
  },
  {
    "interface": "2baf2232-87f7-4ad5-bd80-b6586de08435",
    "ifaddr": {
      "ipv4": "172.17.5.0/24"
    },
    "capacity": {
      "ip": 10
    }
  }
]

According to Huiran Wang, egressIP only works for primary interface on the node.

Version-Release number of selected component (if applicable):

 4.12.0-0.nightly-2022-11-22-012345
RHOS-16.1-RHEL-8-20220804.n.1

How reproducible:

Always

Steps to Reproduce:

Deploy cluster with additional Network on the workers

Actual results:

It is possible to select an egressIP network for a secondary interface

Expected results:

Only primary subnet can be chosen for egressIP

Additional info:

https://issues.redhat.com/browse/OCPQE-12968

• https://github.com/openshift/cloud-network-config-controller/pull/89

When using an install-config with missing VIP values set in the baremetal-platform section, we attempt to get defaults for them by doing a DNS lookup on the cluster domain name. If this lookup fails, we set the error message from DNS as the default value, resulting in a very confusing error message:

[platform.baremetal.apiVIPs: Invalid value: []string{"DNS lookup failure: lookup api.test-cluster.test-domain on 10.0.80.11:53: no such host"}: ip <nil> is invalid, platform.baremetal.apiVIPs: Invalid value: "DNS lookup failure: lookup api.test-cluster.test-domain on 10.0.80.11:53: no such host": "DNS lookup failure: lookup api.test-cluster.test-domain on 10.0.80.11:53: no such host" is not a valid IP, platform.baremetal.apiVIPs: Invalid value: "DNS lookup failure: lookup api.test-cluster.test-domain on 10.0.80.11:53: no such host": IP expected to be in one of the machine networks: 192.168.122.0/23]

This has been the case since the inception of baremetal IPI, but it has gotten considerably worse in 4.12 due to the VIP fields changing from a single string to a list.

If the user doesn't supply a value and we can't generate a sensible default, we should report that the error is that they didn't supply a value, not that they supplied an invalid value that they did not in fact supply:

[platform.baremetal.apiVIPs: Required value: must specify at least one VIP for the API, platform.baremetal.apiVIPs: Required value: must specify VIP for API, when VIP for ingress is set]

• https://github.com/openshift/installer/pull/6486