Reimage ODA

The idea to reimage ODA is to refresh the environment without the need to jump from one by one to reach the last available version, or even rescue the system from S.O. failure/crash. The process to do a reimage can be check in the official documentation but unfortunately can be very tricky because the information (the order and steps) are not 100% clear. The idea is to show you how to reimage using version 18 (18.3 in this example), that represents the last available.

In resume the process is executed in the order:

  1. ILOM: Boot the ISO
  2. Prepare to create the appliance
  3. Upload GI and DB base version to the repository
  4. Linux: Create the appliance
  5. Firmware and patch
  6. Create Oracle Homes and the databases
  7. Finish and clean the install


RMAN, Allocate channel, CDB, and CLOSE: bug

Allocate channel for RMAN is used in various scenarios, most of the time is useful when you use tape as device type or you need to use some kind of format. The way to do the allocation not changed since a long time ago, but when you run the database in container mode you can hit a bug that turns your channel unusable. I will show you the bug and how to avoid it with a simple trick.

This bug hit every version since 12 and I discovered it last year when testing some scenarios, but I was able to test and post just recently. It just occurs for CDB databases and exists just one one-off solution published for 12.2. But there is one workaround more useful and works for every version.

The most interesting part is that everything that we made until now when allocate channel will not work. You can search in all doc available for allocate channel since 9i until 19c the first thing that you made after open the run{} is allocate channel. This is the default and recommended in the docs: for 19c, 18c, and 9i.


Shrink ASM Diskgroup and Exadata Grid Disks

Here I will cover the shrink of ASM diskgroup in Exadata environment running VM’s. The process here is the opposite of what I wrote in the previous post, but have a tricky part that demands attention to avoid errors. The same points that you checked for extending are valid now: number the cells, disks per cell, ASM mirroring, and the VM that you want to change continue to be important, but we have more now. Besides that, the post shows how to verify (and “fix”) if you have something in the ASM internal extent map that can block the shrink.


Increase size for Exadata Grid Disks

A quick article about a maintenance task for Oracle Exadata when you are using OVM and you divided your storage cell disks for every VM. Here I will show you how to extend your Grid Disks to add more space in your ASM diskgroup.

The first thing is being aware of your environment, before everything you need to know the points below because, they are important to calculate the new space, and to avoid do something wrong:

  • Number of cells in your appliance.
  • Number of disks for each cell.
  • Mirroring for your ASM.
  • The VM that you want to add the space.

The “normal” Exadata storage cell has 12 disks, the Extreme Flash version uses 8 disks per storage. If you have doubt about how many disks you have per storage cell, you can connect in each one and check the number of celldisks you have. And before continuing, be aware of Exadata disk division:

To do this change we execute three major steps: ASM, Exadata Storage, and ASM again.


Observer, Quorum

This article closes the series for DG and Fast-Start Failover that I covered with more details the case of isolation can leverage the shutdown of your healthy/running primary database. The “ORA-16830: primary isolated from fast-start failover partners”.

In the first article, I wrote about how one simple detail that impacts dramatically the reliability of your MAA environment. Where you put your Observer in DG environment (when Fast-Start Failover is in use) have a core figure in case of outages, and you can face Primary isolation and shutdown. Besides that, there is no clear documentation to base yourself about “pros and cons” to define the correct place for Observer. You read more in my article here.

In the second article, I wrote about one new feature that can help to have more protected and cover more scenarios for Fast-Start Failover/DG. Using Multiple Observers you can remove the single point of failure and allow you to put one Observer in each side of your environment (primary, standby and a third one). You can read more in my article here.

In this last article I discuss how, even using all the features, there is no perfect solution. Another point is discussing here is how (maybe) Oracle can improve that. Below I will show more details that even multiple observers continue to shutdown a healthy primary database. Unfortunately, it is a lot of tech info and is a log thread output. But you can jump directly to the end to see the discussion about how this can be improved.


Exadata X8, Second look

Every year Oracle arrives and release new version of Exadata with a plenty of new things. We have the natural evolution from hardware (more memory, more cpu…) and sometimes news features from software side. The point for this post today it is not about the HW and SW things, but something is hidden in the small lines of the new X8.


Observer, Where?

Some months ago I got one error with Oracle Data Guard and now I had time to review it again and write this article. Just to be clear since the begin, the discussion here it is not about the error itself, but about the circumstances that generated it.

The environment described here follow, at least, the most common best practices for DG by Oracle. Have 1 dedicated server for each one: Primary Database, Physical Standby Database and Observer. The primary and standby resides in different datacenters in different cities, dedicated network for interconnect between sites, protection mode was Maximum Availability and runs with Fast-Start Failover enabled (with 30 seconds for threshold). The version here is 12.2, but will be the same for 19c. So, nothing so bad in the environment, basic DG configuration trying to follow the best practices.


Reaching Exadata 18c

Here I cover in raw, undocumented and uncommented mode the process to update and upgrade your Exadata using the last version of everything. AND since Oracle 18c was released to use with Oracle Exadata (from SQL Maria), this post include the Oracle 18c upgrade for Grid Infrastructure and Oracle database binaries installation.

Since one friend was decommissioning one old Exadata X2 (running after the end of life), I used to do some tests and here you will find the commands, outputs, images that I use to (in order):

  • Apply the patch for Oracle Grid Infrastructure 12.1.
  • Update the Infiniband switches to last version 2.2.7-1.
  • Apply the patch for Exadata Storage Server using the last version,
  • Apply the patch for Exadata Database Servers using the last version,
  • Upgrade the Oracle Grid Infrastructure 12.1 to Grid Infrastructure 12.2, applying PSU at same time.
  • Upgrade the Oracle Grid Infrastructure 12.2 to Oracle Grid Infrastructure 18c.
  • Install Oracle database binaries for Oracle Database 18c and Create a test Oracle Rac Instance.

Continue lendo…

To infinity and beyond

First, I need to sorry about the long delay since my last update. The last one was in 03/August/2015, so, more than 2 years without news. But I can’t complain about that, was a really busy time here:

  • Oracle Zero Data Loss Recovery Appliance (ZDLRA) deployment: two of them in the end of 2015 and begin of 2016 and working together with two Oracle Exadata to build DR site and reaching RPO and RTO zero. First ZDLRA of Brazil, second in American Latina and one of the first of world to achieve this kind of level.
  • Oracle Open World: linked with the first topic. Because of the size of the project and high complexity of the project I was speaker (together with ZDRLA dev team) in Oracle Open World SFO 2015. Link for presentation here.
  • DR site: specify/implement/deploy/etc DR site during 2015 and 2016, EXA+EXA+DG+ZDLRA+RPO+RTO. So, busy work that unfortunately is not plug and play.
  • Exadata: New deployments to achieve more throughput/performance

For life:

  • Big Move: at the end of 2017 I moved from Brazil to work with a new challenge as DBA in Europe, Luxembourg at eProseed (
  • Family: Family increased, basically 1+1=3 (and this count is correct, don’t worry).

I promise to update more constantly here. Stay tuned!

Aplicando Patch no Exadata

O appliance Oracle Exadata é uma das tecnologias mais modernas para banco de dados Oracle, a união de Hardware e Software. Mas este software precisa ser atualizado de tempos em tempos você precisa aplicar o patch.

O time de Engineered Systems da Oracle disponibiliza os updates para serem aplicadas em toda a pilha: do Exadata Software aos binários do banco de dados. É tarefa do DMA acompanhar isso e deixar seu ambiente atualizado, muitas vezes (aqui no Brasil) estes updates são aplicados pelo time de ACS da Oracle, mas nada impede que você mesmo faça isso.

Este é o foco destes artigos, vou mostrar como proceder com um update completo do Oracle Exadata: Exadata Software, Infiniband, Linux, Update do Grid 11GR2 para 12C e Aplicação de BP nos binários do banco. Eu já descrevi sobre isso no meu blog a uns 3 anos atrás (aqui e aqui), mas muita coisa mudou desde aquela época.


De qualquer forma, antes de começar qualquer update (Oracle Exadata ou não) temos que planejar, verificar qual a versão que estamos e qual queremos aplicar.

Para o Oracle Exadata temos uma nota que contempla tudo isso e que é o início de qualquer update: Exadata Database Machine and Exadata Storage Server Supported Versions (Doc ID 888828.1). Nesta nota estão todas as informações importantes como documentação, as versões existentes, versões recomendadas e matriz de compatibilidade entre Exadata Software e binários do banco.

Então, o primeiro passo é o planejamento, ler e compreender esta nota, principalmente com as versões disponíveis e quais serão aplicadas. Outro ponto fundamental do planejamento é conhecer o seu ambiente, no meu caso a imagem que tenho aqui é a

No tópico Exadata Storage Server 12c da nota identificamos que a última versão e que será aplicada aqui é a (image version e composta pelos seguintes patches e docs:

  • Patch 20748218 – Storage server e InfiniBand
  • Patch 21151982 – Database server (para quem não utiliza OVM)
  • Readme – Note 2014306.1

Na mesma nota temos as versões do Grid Infrastructure e dos bancos de dados que são compatíveis com as imagens do Exadata. Como o meu planejamento inclui atualizar tudo para a última versão estes também serão atualizados.

Através da nota os seguintes updates serão realizados (e demonstrados aqui nestes artigos):

  • Update da imagem para a do Exadata (storage e bancos)
  • Upgrade da versão BP 6 do Grid para a versão BP 9
  • Update da versão BP6 do banco de dados para a versão BP 16

Pensando no appliance Oracle Exadata e na relação entre seus componentes o primeiro ponto a ser atualizado é a própria imagem do Exadata Storage Server (Linux, Exadata Software). Depois temos switchs Infiniband. Na sequência temos o update dos Database Servers e seus componentes (Grid e os binários do banco de dados).

Antes de iniciar o update precisamos montar o plano de atualização, qual a ordem de instalação dos patches. Isso só pode ser feito lendo todos os Readmes dos patches que vamos aplicar. Lendo o Readme do Patch 20748218 e Patch 21151982 descobrimos que existem dois métodos possíveis para atualizar o Database Server.

O primeiro é através da imagem ISO que acompanha o Patch 21151982 e o segundo é através da criação do repositório Exadata local. Este último método é o que utilizarei aqui, ele tem algumas vantagens que descreverei em detalhes depois mas é importante entender como funciona. Basicamente você cria um repositório local com os rpm’s que serão aplicados (os mesmo presentes na ISO). Infelizmente esta criação pode demorar um pouco e por isso será o primeiro procedimento a ser realizado.

Através da leitura de todas as notas do MOS envolvidas e de todos os Readmes dos patches aplicados o seguinte plano foi criado:

  1. Criação do repositório Exadata para Database Server
  2. Update do Storage Server e Switch Infiniband
  3. Update do Database Server
  4. Upgrade do GRID para 12C e Instalação dos binários 12C
  5. Update dos binários para o BP16

Abaixo a descrição detalhas de cada uma destas etapas acima listadas.

A última atualização deste artigo foi em 03/08/2015.