Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

buffer: prevent cache corruption #7808

Merged
merged 1 commit into from
Jun 30, 2023
Merged

buffer: prevent cache corruption #7808

merged 1 commit into from
Jun 30, 2023

Conversation

lyakh
Copy link
Collaborator

@lyakh lyakh commented Jun 15, 2023

Recent research discovered a set of potential issues related with cache prefetch. Specifically it seems like uncached access to memory can cause cache prefetch. This can cause problems in buffer_attach() and buffer_detach() where buffers are added to or removed from lists respectively via uncached addresses, after which they can be used via cached addresses. Add proper cache synchronisation and interrupt locking to protect against such memory corruption.

Recent research discovered a set of potential issues related with
cache prefetch. Specifically it seems like uncached access to memory
can cause cache prefetch. This can cause problems in buffer_attach()
and buffer_detach() where buffers are added to or removed from lists
respectively via uncached addresses, after which they can be used via
cached addresses. Add proper cache synchronisation and interrupt
locking to protect against such memory corruption.

Signed-off-by: Guennadi Liakhovetski <[email protected]>
@btian1
Copy link
Contributor

btian1 commented Jun 16, 2023

"Recent research discovered a set of potential issues related with cache prefetch. Specifically it seems like uncached access to memory can cause cache prefetch. "

is this reasonable? if access memory in uncached, then dsp should perform cache prefetch, right?

@lyakh
Copy link
Collaborator Author

lyakh commented Jun 16, 2023

is this reasonable? if access memory in uncached, then dsp should perform cache prefetch, right?

@btian1 #7191 (comment)

@btian1
Copy link
Contributor

btian1 commented Jun 19, 2023

I would suggest wait for some time for #7191 , if still reproducible, we can ask keqiao help for reproduce with this PR.

src/audio/buffer.c Show resolved Hide resolved
if (!buffer) {
comp_err(dev, "module_adapter_prepare(): failed to allocate local buffer");
ret = -ENOMEM;
goto free;
}

irq_local_disable(flags);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when turn off irq, means some or one of irq have negative impact with buffer_attach.
this part of change can make buffer_attach more secure, just not sure it is must or not.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is needed because we're processing an IPC here and we might get interrupted by the scheduler, already running a pipeline to which we're attaching an additional source or sink.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does feel out-of-place in module_adapter.c level code. Given both IPC and pipeline are run in regular threads, we should be able to use plain mutex'es for exclusive access here. This would make it also userspace compatible for future-proofing this code.

But maybe beyond this PR, especially given pipeline_connect() already does the IRQ enable/disable dance, so the same should be done here to be safe.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does feel out-of-place in module_adapter.c level code. Given both IPC and pipeline are run in regular threads, we should be able to use plain mutex'es for exclusive access here. This would make it also userspace compatible for future-proofing this code.

@kv2019i maybe mutexes would work too, but would take context switching:

  1. binding of an additional pipeline is taking place
  2. a timer interrupt arrives
  3. the copying is scheduled
  4. since the LL scheduler has a higher priority than the IPC thread we switch to copying
  5. we try to take the mutex but it's held by the IPC thread
  6. the copying thread is suspended and the IPC thread is scheduled (priority inversion)
  7. the IPC thread finishes binding and releases the mutex
  8. we switch back to the copying thread

So, this should work, but would involve a bunch of switching. If the protected area is small enough, locking interrupts should be simple enough. But if we shouldn't use priority calls, then we can use a mutex

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lyakh Ack. Let's revisit this when we introduce user-space, this is anyways not the only place where we have explicit irq control in the SOF audio code.

Copy link
Collaborator

@kv2019i kv2019i left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, took a long time to review, but this is a complex piece of code to review. I'm leaning towards approving as this does problems in current codebase. See more comments inline.

src/audio/buffer.c Show resolved Hide resolved
if (!buffer) {
comp_err(dev, "module_adapter_prepare(): failed to allocate local buffer");
ret = -ENOMEM;
goto free;
}

irq_local_disable(flags);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does feel out-of-place in module_adapter.c level code. Given both IPC and pipeline are run in regular threads, we should be able to use plain mutex'es for exclusive access here. This would make it also userspace compatible for future-proofing this code.

But maybe beyond this PR, especially given pipeline_connect() already does the IRQ enable/disable dance, so the same should be done here to be safe.

@kv2019i
Copy link
Collaborator

kv2019i commented Jun 30, 2023

@kv2019i kv2019i merged commit 3e3d0cd into thesofproject:main Jun 30, 2023
@lyakh lyakh deleted the buffer branch June 30, 2023 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants