[chromium-discuss] The minidump can not always generated by breakpad when there is a crash occurred.

[Liu Xiao]

*[Repetition steps]:*

1. keep breakpad enabled;

2. use chrome to browse webpage;

3. repeat step 2 until there is a crash occurred.



*[**What is the expected?**]:*

minidump file will be always generated successfully.



*[What went wrong?]:*

Sometimes minidump file can not be generated,and we can see

"google-breakpad: Failed to generate minidump." from the log.



*[Probability]:*

However,the probability of this problem is low, it is difficult to reappear.



*[Question]:*

Is it an inherent defect of breakpad ?

What is the root cause of it, and has it been resolved in the new version
of chromium ?



Thank you very much.

[Liu Xiao]

*[Repetition steps]:*

1. keep breakpad enabled;

2. use chrome to browse webpage;

3. repeat step 2 until there is a crash occurred.



*[**What is the expected?**]:*

minidump file will be always generated successfully.



*[What went wrong?]:*

Sometimes minidump file can not be generated,and we can see

"google-breakpad: Failed to generate minidump." from the log.



*[Probability]:*

However,the probability of this problem is low, it is difficult to reappear.



*[Question]:*

Is it an inherent defect of breakpad ?

What is the root cause of it, and has it been resolved in the new version
of chromium ?



Thank you very much.

[Liu Xiao]

Version : 63.0.3239.111.

[Liu Xiao]

Version : 63.0.3239.111.

[Torne (Richard Coles)]

On Android the process is usually 32-bit, which means address space is very
limited. When the address space is exhausted then you'll get an OOM crash -
and if it's bad enough, breakpad will also fail to function correctly
because it will not be able to find any free address space to mmap() the
things it needs to look at to generate a proper crash report.

If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Post by Liu Xiao
Version : 63.0.3239.111.
--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Liu Xiao]

Thanks for your answer.

If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context.

Post by Torne (Richard Coles)

It is very likely that the problem is derived from an OOM crash.
From the source code, we can see when crashed the crashed process call

system function 'clone' to create a child-process which is responsible for
minidump.

Post by Torne (Richard Coles)

Under this mode (not full out-of-process), the situation "runs inside

the crashed process in a compromised context" will also exist ?

=========================================================================
bool ExceptionHandler::GenerateDump(CrashContext *context) {
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg, NULL,
NULL,
NULL);
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);
SendContinueSignalToChild();
int status;
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));

sys_close(fdes[1]);
}
=========================================================================

On Wednesday, July 18, 2018 at 5:18:15 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
On Android the process is usually 32-bit, which means address space is
very limited. When the address space is exhausted then you'll get an OOM
crash - and if it's bad enough, breakpad will also fail to function
correctly because it will not be able to find any free address space to
mmap() the things it needs to look at to generate a proper crash report.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Version : 63.0.3239.111.
--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Torne (Richard Coles)]

This sys_clone() call is basically fork() - the child process is mostly
identical to the parent process, including having all the same address
space mappings and not having any free virtual address space as a result.
The context is still compromised, because it's the same context.

The only way to be running in a safe environment in the crash handler is to
have the dump generation run inside another process that was already
running before the crash, or to exec() another binary after forking to
actually get a fresh environment, and breakpad cannot do either of those
things.

OOM crashes on 32-bit android aren't very interesting in general - address
space is very limited and it's quite possible to run out without anything
actually being wrong; some devices also have bad graphics drivers that leak
address space. It's unlikely you can do anything to address these crashes
even if you had a dump.

Post by Liu Xiao
Thanks for your answer.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context.

Post by Torne (Richard Coles)

It is very likely that the problem is derived from an OOM crash.
From the source code, we can see when crashed the crashed process call

system function 'clone' to create a child-process which is responsible for
minidump.

Post by Torne (Richard Coles)

Under this mode (not full out-of-process), the situation "runs inside

the crashed process in a compromised context" will also exist ?
=========================================================================
bool ExceptionHandler::GenerateDump(CrashContext *context) {
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg, NULL,
NULL,
NULL);
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);
SendContinueSignalToChild();
int status;
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));
sys_close(fdes[1]);
}
=========================================================================
On Wednesday, July 18, 2018 at 5:18:15 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
On Android the process is usually 32-bit, which means address space is
very limited. When the address space is exhausted then you'll get an OOM
crash - and if it's bad enough, breakpad will also fail to function
correctly because it will not be able to find any free address space to
mmap() the things it needs to look at to generate a proper crash report.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Version : 63.0.3239.111.

Post by Torne (Richard Coles)

--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Liu Xiao]

Thank you very much.

his sys_clone() call is basically fork() - the child process is mostly
identical to the parent process.

Post by Torne (Richard Coles)

Yes, creating child-process by sys_clone() is not the full

out-of-process mode.

The only way to be running in a safe environment in the crash handler is to
have the dump generation run inside another process.

Post by Torne (Richard Coles)

Yes, I have got it from the breakpad doc, that is called out-of-process.
I want to implement it before, but it is sensitive for creating a new

process, and I am not sure if it will resolve the problem in OOM Crash,
what do you think ?

OOM crashes on 32-bit android aren't very interesting in general

Post by Torne (Richard Coles)

Do you mean OOM Crash is a bothersome problem on 32-bit Android?
I am using Android 7.0, it is definitely a 32-bit Android, or there may

be a 64-bit version ? How to comfirm it ?

On Thursday, July 19, 2018 at 12:04:22 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
This sys_clone() call is basically fork() - the child process is mostly
identical to the parent process, including having all the same address
space mappings and not having any free virtual address space as a result.
The context is still compromised, because it's the same context.
The only way to be running in a safe environment in the crash handler is
to have the dump generation run inside another process that was already
running before the crash, or to exec() another binary after forking to
actually get a fresh environment, and breakpad cannot do either of those
things.
OOM crashes on 32-bit android aren't very interesting in general - address
space is very limited and it's quite possible to run out without anything
actually being wrong; some devices also have bad graphics drivers that leak
address space. It's unlikely you can do anything to address these crashes
even if you had a dump.

Thanks for your answer.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context.

Post by Torne (Richard Coles)

It is very likely that the problem is derived from an OOM crash.
From the source code, we can see when crashed the crashed process call

system function 'clone' to create a child-process which is responsible for
minidump.

Post by Torne (Richard Coles)

Under this mode (not full out-of-process), the situation "runs inside

the crashed process in a compromised context" will also exist ?
=========================================================================
bool ExceptionHandler::GenerateDump(CrashContext *context) {
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg, NULL,
NULL,
NULL);
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);
SendContinueSignalToChild();
int status;
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));
sys_close(fdes[1]);
}
=========================================================================
On Wednesday, July 18, 2018 at 5:18:15 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
On Android the process is usually 32-bit, which means address space is
very limited. When the address space is exhausted then you'll get an OOM
crash - and if it's bad enough, breakpad will also fail to function
correctly because it will not be able to find any free address space to
mmap() the things it needs to look at to generate a proper crash report.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Version : 63.0.3239.111.

Post by Torne (Richard Coles)

--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Torne (Richard Coles)]

Post by Liu Xiao
Thank you very much.
his sys_clone() call is basically fork() - the child process is mostly
identical to the parent process.

Post by Torne (Richard Coles)

Yes, creating child-process by sys_clone() is not the full

out-of-process mode.

I'm not sure what you're referring to here.

Post by Liu Xiao
The only way to be running in a safe environment in the crash handler is
to have the dump generation run inside another process.

Post by Torne (Richard Coles)

Yes, I have got it from the breakpad doc, that is called out-of-process.

I'm not aware of breakpad having any other way to operate - we're replacing
it with crashpad in part to avoid this limitation.

Post by Liu Xiao

Post by Torne (Richard Coles)

I want to implement it before, but it is sensitive for creating a new

process, and I am not sure if it will resolve the problem in OOM Crash,
what do you think ?

I'm not sure what you're asking; as I said we don't use/support any other
way to handle crashes on android right now (as we haven't yet moved to
crashpad) and I'm not aware of breakpad having any alternative.

Post by Liu Xiao
OOM crashes on 32-bit android aren't very interesting in general

Post by Torne (Richard Coles)

Do you mean OOM Crash is a bothersome problem on 32-bit Android?

No, I mean that it's not generally considered important to investigate OOM
crashes because they very rarely lead to finding an actual bug that can be
fixed - so we don't consider it a huge problem that breakpad fails to
generate a dump in some OOM situations. (though it is still a bit of a
problem, and we're moving to crashpad in part to avoid it).

Post by Liu Xiao

Post by Torne (Richard Coles)

I am using Android 7.0, it is definitely a 32-bit Android, or there may

be a 64-bit version ? How to comfirm it ?

It's not about what version of Android you are running - what matters is
whether the chromium build you are using is 32-bit or 64-bit. By default
it's 32-bit, and while it's possible to build a 64-bit build for some
configurations (though not all; monochrome is always 32-bit for example) we
don't ship the 64-bit version to anyone and generally don't recommend that
people use it, because it currently uses quite a lot more memory, to the
point that the user experience is significantly worse. Unless you
explicitly chose to build the 64-bit version when configuring your build,
you are using the 32-bit version.

Post by Liu Xiao
On Thursday, July 19, 2018 at 12:04:22 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
This sys_clone() call is basically fork() - the child process is mostly
identical to the parent process, including having all the same address
space mappings and not having any free virtual address space as a result.
The context is still compromised, because it's the same context.
The only way to be running in a safe environment in the crash handler is
to have the dump generation run inside another process that was already
running before the crash, or to exec() another binary after forking to
actually get a fresh environment, and breakpad cannot do either of those
things.
OOM crashes on 32-bit android aren't very interesting in general -
address space is very limited and it's quite possible to run out without
anything actually being wrong; some devices also have bad graphics drivers
that leak address space. It's unlikely you can do anything to address these
crashes even if you had a dump.

Thanks for your answer.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context.

Post by Torne (Richard Coles)

It is very likely that the problem is derived from an OOM crash.
From the source code, we can see when crashed the crashed process

call system function 'clone' to create a child-process which is responsible
for minidump.

Post by Torne (Richard Coles)

Under this mode (not full out-of-process), the situation "runs inside

the crashed process in a compromised context" will also exist ?
=========================================================================
bool ExceptionHandler::GenerateDump(CrashContext *context) {
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg, NULL,
NULL,
NULL);
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);
SendContinueSignalToChild();
int status;
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));
sys_close(fdes[1]);
}
=========================================================================
On Wednesday, July 18, 2018 at 5:18:15 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
On Android the process is usually 32-bit, which means address space is
very limited. When the address space is exhausted then you'll get an OOM
crash - and if it's bad enough, breakpad will also fail to function
correctly because it will not be able to find any free address space to
mmap() the things it needs to look at to generate a proper crash report.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Version : 63.0.3239.111.

Post by Torne (Richard Coles)

--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Liu Xiao]

Thank you very much.

I want to ask a question that when crashpad can be implemented on Linux and
Android ? And it is sure to resolve the problem "The minidump can not
always generated by breakpad" ?

Post by Torne (Richard Coles)

Post by Liu Xiao
Thank you very much.
his sys_clone() call is basically fork() - the child process is mostly
identical to the parent process.

Post by Torne (Richard Coles)

Yes, creating child-process by sys_clone() is not the full

out-of-process mode.

I'm not sure what you're referring to here.

Post by Liu Xiao
The only way to be running in a safe environment in the crash handler is
to have the dump generation run inside another process.

Post by Torne (Richard Coles)

Yes, I have got it from the breakpad doc, that is called

out-of-process.

I'm not aware of breakpad having any other way to operate - we're
replacing it with crashpad in part to avoid this limitation.

Post by Liu Xiao

Post by Torne (Richard Coles)

I want to implement it before, but it is sensitive for creating a new

process, and I am not sure if it will resolve the problem in OOM Crash,
what do you think ?

I'm not sure what you're asking; as I said we don't use/support any other
way to handle crashes on android right now (as we haven't yet moved to
crashpad) and I'm not aware of breakpad having any alternative.

Post by Liu Xiao
OOM crashes on 32-bit android aren't very interesting in general

Post by Torne (Richard Coles)

Do you mean OOM Crash is a bothersome problem on 32-bit Android?

No, I mean that it's not generally considered important to investigate OOM
crashes because they very rarely lead to finding an actual bug that can be
fixed - so we don't consider it a huge problem that breakpad fails to
generate a dump in some OOM situations. (though it is still a bit of a
problem, and we're moving to crashpad in part to avoid it).

Post by Liu Xiao

Post by Torne (Richard Coles)

I am using Android 7.0, it is definitely a 32-bit Android, or there

may be a 64-bit version ? How to comfirm it ?

It's not about what version of Android you are running - what matters is
whether the chromium build you are using is 32-bit or 64-bit. By default
it's 32-bit, and while it's possible to build a 64-bit build for some
configurations (though not all; monochrome is always 32-bit for example) we
don't ship the 64-bit version to anyone and generally don't recommend that
people use it, because it currently uses quite a lot more memory, to the
point that the user experience is significantly worse. Unless you
explicitly chose to build the 64-bit version when configuring your build,
you are using the 32-bit version.

Post by Liu Xiao
On Thursday, July 19, 2018 at 12:04:22 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
This sys_clone() call is basically fork() - the child process is mostly
identical to the parent process, including having all the same address
space mappings and not having any free virtual address space as a result.
The context is still compromised, because it's the same context.
The only way to be running in a safe environment in the crash handler is
to have the dump generation run inside another process that was already
running before the crash, or to exec() another binary after forking to
actually get a fresh environment, and breakpad cannot do either of those
things.
OOM crashes on 32-bit android aren't very interesting in general -
address space is very limited and it's quite possible to run out without
anything actually being wrong; some devices also have bad graphics drivers
that leak address space. It's unlikely you can do anything to address these
crashes even if you had a dump.

Thanks for your answer.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context.

Post by Torne (Richard Coles)

It is very likely that the problem is derived from an OOM crash.
From the source code, we can see when crashed the crashed process

call system function 'clone' to create a child-process which is responsible
for minidump.

Post by Torne (Richard Coles)

Under this mode (not full out-of-process), the situation "runs

inside the crashed process in a compromised context" will also exist ?
=========================================================================
bool ExceptionHandler::GenerateDump(CrashContext *context) {
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg, NULL,
NULL,
NULL);
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);
SendContinueSignalToChild();
int status;
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));
sys_close(fdes[1]);
}
=========================================================================
On Wednesday, July 18, 2018 at 5:18:15 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
On Android the process is usually 32-bit, which means address space is
very limited. When the address space is exhausted then you'll get an OOM
crash - and if it's bad enough, breakpad will also fail to function
correctly because it will not be able to find any free address space to
mmap() the things it needs to look at to generate a proper crash report.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Version : 63.0.3239.111.

Post by Torne (Richard Coles)

--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Torne (Richard Coles)]

Crashpad for android is currently being worked on and hopefully will be
enabled in the near future. I don't know about the status on Linux.

No crash handling solution originating from inside the app can guarantee
that a dump will always be generated perfectly, but crashpad runs in a
non-compromised context which makes the current most common failure modes
of breakpad impossible.

Post by Liu Xiao
Thank you very much.
I want to ask a question that when crashpad can be implemented on Linux
and Android ? And it is sure to resolve the problem "The minidump can not
always generated by breakpad" ?

Post by Torne (Richard Coles)

Post by Liu Xiao
Thank you very much.
his sys_clone() call is basically fork() - the child process is mostly
identical to the parent process.

Post by Torne (Richard Coles)

Yes, creating child-process by sys_clone() is not the full

out-of-process mode.

I'm not sure what you're referring to here.

Post by Liu Xiao
The only way to be running in a safe environment in the crash handler is
to have the dump generation run inside another process.

Post by Torne (Richard Coles)

Yes, I have got it from the breakpad doc, that is called

out-of-process.

I'm not aware of breakpad having any other way to operate - we're
replacing it with crashpad in part to avoid this limitation.

Post by Liu Xiao

Post by Torne (Richard Coles)

I want to implement it before, but it is sensitive for creating a

new process, and I am not sure if it will resolve the problem in OOM Crash,
what do you think ?

I'm not sure what you're asking; as I said we don't use/support any other
way to handle crashes on android right now (as we haven't yet moved to
crashpad) and I'm not aware of breakpad having any alternative.

Post by Liu Xiao
OOM crashes on 32-bit android aren't very interesting in general

Post by Torne (Richard Coles)

Do you mean OOM Crash is a bothersome problem on 32-bit Android?

No, I mean that it's not generally considered important to investigate
OOM crashes because they very rarely lead to finding an actual bug that can
be fixed - so we don't consider it a huge problem that breakpad fails to
generate a dump in some OOM situations. (though it is still a bit of a
problem, and we're moving to crashpad in part to avoid it).

Post by Liu Xiao

Post by Torne (Richard Coles)

I am using Android 7.0, it is definitely a 32-bit Android, or there

may be a 64-bit version ? How to comfirm it ?

It's not about what version of Android you are running - what matters is
whether the chromium build you are using is 32-bit or 64-bit. By default
it's 32-bit, and while it's possible to build a 64-bit build for some
configurations (though not all; monochrome is always 32-bit for example) we
don't ship the 64-bit version to anyone and generally don't recommend that
people use it, because it currently uses quite a lot more memory, to the
point that the user experience is significantly worse. Unless you
explicitly chose to build the 64-bit version when configuring your build,
you are using the 32-bit version.

Post by Liu Xiao
On Thursday, July 19, 2018 at 12:04:22 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
This sys_clone() call is basically fork() - the child process is mostly
identical to the parent process, including having all the same address
space mappings and not having any free virtual address space as a result.
The context is still compromised, because it's the same context.
The only way to be running in a safe environment in the crash handler
is to have the dump generation run inside another process that was already
running before the crash, or to exec() another binary after forking to
actually get a fresh environment, and breakpad cannot do either of those
things.
OOM crashes on 32-bit android aren't very interesting in general -
address space is very limited and it's quite possible to run out without
anything actually being wrong; some devices also have bad graphics drivers
that leak address space. It's unlikely you can do anything to address these
crashes even if you had a dump.

Thanks for your answer.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context.

Post by Torne (Richard Coles)

It is very likely that the problem is derived from an OOM crash.
From the source code, we can see when crashed the crashed process

call system function 'clone' to create a child-process which is responsible
for minidump.

Post by Torne (Richard Coles)

Under this mode (not full out-of-process), the situation "runs

inside the crashed process in a compromised context" will also exist ?
=========================================================================
bool ExceptionHandler::GenerateDump(CrashContext *context) {
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg,
NULL, NULL,
NULL);
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);
SendContinueSignalToChild();
int status;
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));
sys_close(fdes[1]);
}
=========================================================================
On Wednesday, July 18, 2018 at 5:18:15 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
On Android the process is usually 32-bit, which means address space
is very limited. When the address space is exhausted then you'll get an OOM
crash - and if it's bad enough, breakpad will also fail to function
correctly because it will not be able to find any free address space to
mmap() the things it needs to look at to generate a proper crash report.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Version : 63.0.3239.111.

Post by Torne (Richard Coles)

--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Liu Xiao]

Look forward to crashpad release.

Our team is now mainly using "unwind the stack (using conventional
libunwind)" for analyzing crash, and it is rarely failed.
Do you think it is a safe method for developer?

Post by Torne (Richard Coles)
Crashpad for android is currently being worked on and hopefully will be
enabled in the near future. I don't know about the status on Linux.
No crash handling solution originating from inside the app can guarantee
that a dump will always be generated perfectly, but crashpad runs in a
non-compromised context which makes the current most common failure modes
of breakpad impossible.

Post by Liu Xiao
Thank you very much.
I want to ask a question that when crashpad can be implemented on Linux
and Android ? And it is sure to resolve the problem "The minidump can not
always generated by breakpad" ?

Post by Torne (Richard Coles)

Post by Liu Xiao
Thank you very much.
his sys_clone() call is basically fork() - the child process is mostly
identical to the parent process.

Post by Torne (Richard Coles)

Yes, creating child-process by sys_clone() is not the full

out-of-process mode.

I'm not sure what you're referring to here.

Post by Liu Xiao
The only way to be running in a safe environment in the crash handler
is to have the dump generation run inside another process.

Post by Torne (Richard Coles)

Yes, I have got it from the breakpad doc, that is called

out-of-process.

I'm not aware of breakpad having any other way to operate - we're
replacing it with crashpad in part to avoid this limitation.

Post by Liu Xiao

Post by Torne (Richard Coles)

I want to implement it before, but it is sensitive for creating a

new process, and I am not sure if it will resolve the problem in OOM Crash,
what do you think ?

I'm not sure what you're asking; as I said we don't use/support any
other way to handle crashes on android right now (as we haven't yet moved
to crashpad) and I'm not aware of breakpad having any alternative.

Post by Liu Xiao
OOM crashes on 32-bit android aren't very interesting in general

Post by Torne (Richard Coles)

Do you mean OOM Crash is a bothersome problem on 32-bit Android?

No, I mean that it's not generally considered important to investigate
OOM crashes because they very rarely lead to finding an actual bug that can
be fixed - so we don't consider it a huge problem that breakpad fails to
generate a dump in some OOM situations. (though it is still a bit of a
problem, and we're moving to crashpad in part to avoid it).

Post by Liu Xiao

Post by Torne (Richard Coles)

I am using Android 7.0, it is definitely a 32-bit Android, or there

may be a 64-bit version ? How to comfirm it ?

It's not about what version of Android you are running - what matters is
whether the chromium build you are using is 32-bit or 64-bit. By default
it's 32-bit, and while it's possible to build a 64-bit build for some
configurations (though not all; monochrome is always 32-bit for example) we
don't ship the 64-bit version to anyone and generally don't recommend that
people use it, because it currently uses quite a lot more memory, to the
point that the user experience is significantly worse. Unless you
explicitly chose to build the 64-bit version when configuring your build,
you are using the 32-bit version.

Post by Liu Xiao
On Thursday, July 19, 2018 at 12:04:22 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
This sys_clone() call is basically fork() - the child process is
mostly identical to the parent process, including having all the same
address space mappings and not having any free virtual address space as a
result. The context is still compromised, because it's the same context.
The only way to be running in a safe environment in the crash handler
is to have the dump generation run inside another process that was already
running before the crash, or to exec() another binary after forking to
actually get a fresh environment, and breakpad cannot do either of those
things.
OOM crashes on 32-bit android aren't very interesting in general -
address space is very limited and it's quite possible to run out without
anything actually being wrong; some devices also have bad graphics drivers
that leak address space. It's unlikely you can do anything to address these
crashes even if you had a dump.

Thanks for your answer.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context.

Post by Torne (Richard Coles)

It is very likely that the problem is derived from an OOM crash.
From the source code, we can see when crashed the crashed process

call system function 'clone' to create a child-process which is responsible
for minidump.

Post by Torne (Richard Coles)

Under this mode (not full out-of-process), the situation "runs

inside the crashed process in a compromised context" will also exist ?
=========================================================================
bool ExceptionHandler::GenerateDump(CrashContext *context) {
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg,
NULL, NULL,
NULL);
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);
SendContinueSignalToChild();
int status;
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));
sys_close(fdes[1]);
}
=========================================================================
On Wednesday, July 18, 2018 at 5:18:15 AM UTC+8, Torne (Richard

Post by Torne (Richard Coles)
On Android the process is usually 32-bit, which means address space
is very limited. When the address space is exhausted then you'll get an OOM
crash - and if it's bad enough, breakpad will also fail to function
correctly because it will not be able to find any free address space to
mmap() the things it needs to look at to generate a proper crash report.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Version : 63.0.3239.111.

Post by Torne (Richard Coles)

--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Torne (Richard Coles)]

Post by Liu Xiao
Look forward to crashpad release.
Our team is now mainly using "unwind the stack (using conventional
libunwind)" for analyzing crash, and it is rarely failed.
Do you think it is a safe method for developer?

In our official builds it's impossible to use libunwind, because official
builds don't include unwind tables in the binary on-device, to reduce the
size of the binary.

If you're including the unwind tables then libunwind will probably
generally work but I have no idea whether that's actually
expected/guaranteed to be safe to call in a compromised context.

Post by Liu Xiao

Post by Torne (Richard Coles)
Crashpad for android is currently being worked on and hopefully will be
enabled in the near future. I don't know about the status on Linux.
No crash handling solution originating from inside the app can guarantee
that a dump will always be generated perfectly, but crashpad runs in a
non-compromised context which makes the current most common failure modes
of breakpad impossible.

Post by Liu Xiao
Thank you very much.
I want to ask a question that when crashpad can be implemented on Linux
and Android ? And it is sure to resolve the problem "The minidump can not
always generated by breakpad" ?

Post by Torne (Richard Coles)

Post by Liu Xiao
Thank you very much.
his sys_clone() call is basically fork() - the child process is mostly
identical to the parent process.

Post by Torne (Richard Coles)

Yes, creating child-process by sys_clone() is not the full

out-of-process mode.

I'm not sure what you're referring to here.

Post by Liu Xiao
The only way to be running in a safe environment in the crash handler
is to have the dump generation run inside another process.

Post by Torne (Richard Coles)

Yes, I have got it from the breakpad doc, that is called

out-of-process.

I'm not aware of breakpad having any other way to operate - we're
replacing it with crashpad in part to avoid this limitation.

Post by Liu Xiao

Post by Torne (Richard Coles)

I want to implement it before, but it is sensitive for creating a

new process, and I am not sure if it will resolve the problem in OOM Crash,
what do you think ?

I'm not sure what you're asking; as I said we don't use/support any
other way to handle crashes on android right now (as we haven't yet moved
to crashpad) and I'm not aware of breakpad having any alternative.

Post by Liu Xiao
OOM crashes on 32-bit android aren't very interesting in general

Post by Torne (Richard Coles)

Do you mean OOM Crash is a bothersome problem on 32-bit Android?

No, I mean that it's not generally considered important to investigate
OOM crashes because they very rarely lead to finding an actual bug that can
be fixed - so we don't consider it a huge problem that breakpad fails to
generate a dump in some OOM situations. (though it is still a bit of a
problem, and we're moving to crashpad in part to avoid it).

Post by Liu Xiao

Post by Torne (Richard Coles)

I am using Android 7.0, it is definitely a 32-bit Android, or there

may be a 64-bit version ? How to comfirm it ?

It's not about what version of Android you are running - what matters
is whether the chromium build you are using is 32-bit or 64-bit. By default
it's 32-bit, and while it's possible to build a 64-bit build for some
configurations (though not all; monochrome is always 32-bit for example) we
don't ship the 64-bit version to anyone and generally don't recommend that
people use it, because it currently uses quite a lot more memory, to the
point that the user experience is significantly worse. Unless you
explicitly chose to build the 64-bit version when configuring your build,
you are using the 32-bit version.

Post by Liu Xiao
On Thursday, July 19, 2018 at 12:04:22 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
This sys_clone() call is basically fork() - the child process is
mostly identical to the parent process, including having all the same
address space mappings and not having any free virtual address space as a
result. The context is still compromised, because it's the same context.
The only way to be running in a safe environment in the crash handler
is to have the dump generation run inside another process that was already
running before the crash, or to exec() another binary after forking to
actually get a fresh environment, and breakpad cannot do either of those
things.
OOM crashes on 32-bit android aren't very interesting in general -
address space is very limited and it's quite possible to run out without
anything actually being wrong; some devices also have bad graphics drivers
that leak address space. It's unlikely you can do anything to address these
crashes even if you had a dump.

Thanks for your answer.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context.

Post by Torne (Richard Coles)

It is very likely that the problem is derived from an OOM crash.
From the source code, we can see when crashed the crashed process

call system function 'clone' to create a child-process which is responsible
for minidump.

Post by Torne (Richard Coles)

Under this mode (not full out-of-process), the situation "runs

inside the crashed process in a compromised context" will also exist ?
=========================================================================
bool ExceptionHandler::GenerateDump(CrashContext *context) {
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg,
NULL, NULL,
NULL);
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);
SendContinueSignalToChild();
int status;
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));
sys_close(fdes[1]);
}
=========================================================================
On Wednesday, July 18, 2018 at 5:18:15 AM UTC+8, Torne (Richard

Post by Torne (Richard Coles)
On Android the process is usually 32-bit, which means address space
is very limited. When the address space is exhausted then you'll get an OOM
crash - and if it's bad enough, breakpad will also fail to function
correctly because it will not be able to find any free address space to
mmap() the things it needs to look at to generate a proper crash report.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Version : 63.0.3239.111.

Post by Torne (Richard Coles)

--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Liu Xiao]

Thank you very much.

In our official builds it's impossible to use libunwind, because official
builds don't include unwind tables in the binary on-device, to reduce the
size of the binary.

Post by Torne (Richard Coles)

Could I understand like this : the minidump generated by breakpad is the

only method for analyzing crash on official builds, though there is the
defect "The minidump can not always generated" ?

On Wednesday, July 25, 2018 at 11:12:54 PM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)

Look forward to crashpad release.
Our team is now mainly using "unwind the stack (using conventional
libunwind)" for analyzing crash, and it is rarely failed.
Do you think it is a safe method for developer?

In our official builds it's impossible to use libunwind, because official
builds don't include unwind tables in the binary on-device, to reduce the
size of the binary.
If you're including the unwind tables then libunwind will probably
generally work but I have no idea whether that's actually
expected/guaranteed to be safe to call in a compromised context.
On Tuesday, July 24, 2018 at 11:01:44 PM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
Crashpad for android is currently being worked on and hopefully will be
enabled in the near future. I don't know about the status on Linux.
No crash handling solution originating from inside the app can guarantee
that a dump will always be generated perfectly, but crashpad runs in a
non-compromised context which makes the current most common failure modes
of breakpad impossible.

Post by Liu Xiao
Thank you very much.
I want to ask a question that when crashpad can be implemented on Linux
and Android ? And it is sure to resolve the problem "The minidump can not
always generated by breakpad" ?

Post by Torne (Richard Coles)

Post by Liu Xiao
Thank you very much.
his sys_clone() call is basically fork() - the child process is
mostly identical to the parent process.

Post by Torne (Richard Coles)

Yes, creating child-process by sys_clone() is not the full

out-of-process mode.

I'm not sure what you're referring to here.

Post by Liu Xiao
The only way to be running in a safe environment in the crash handler
is to have the dump generation run inside another process.

Post by Torne (Richard Coles)

Yes, I have got it from the breakpad doc, that is called

out-of-process.

I'm not aware of breakpad having any other way to operate - we're
replacing it with crashpad in part to avoid this limitation.

Post by Liu Xiao

Post by Torne (Richard Coles)

I want to implement it before, but it is sensitive for creating a

new process, and I am not sure if it will resolve the problem in OOM Crash,
what do you think ?

I'm not sure what you're asking; as I said we don't use/support any
other way to handle crashes on android right now (as we haven't yet moved
to crashpad) and I'm not aware of breakpad having any alternative.

Post by Liu Xiao
OOM crashes on 32-bit android aren't very interesting in general

Post by Torne (Richard Coles)

Do you mean OOM Crash is a bothersome problem on 32-bit Android?

No, I mean that it's not generally considered important to investigate
OOM crashes because they very rarely lead to finding an actual bug that can
be fixed - so we don't consider it a huge problem that breakpad fails to
generate a dump in some OOM situations. (though it is still a bit of a
problem, and we're moving to crashpad in part to avoid it).

Post by Liu Xiao

Post by Torne (Richard Coles)

I am using Android 7.0, it is definitely a 32-bit Android, or

there may be a 64-bit version ? How to comfirm it ?

It's not about what version of Android you are running - what matters
is whether the chromium build you are using is 32-bit or 64-bit. By default
it's 32-bit, and while it's possible to build a 64-bit build for some
configurations (though not all; monochrome is always 32-bit for example) we
don't ship the 64-bit version to anyone and generally don't recommend that
people use it, because it currently uses quite a lot more memory, to the
point that the user experience is significantly worse. Unless you
explicitly chose to build the 64-bit version when configuring your build,
you are using the 32-bit version.

Post by Liu Xiao
On Thursday, July 19, 2018 at 12:04:22 AM UTC+8, Torne (Richard

Post by Torne (Richard Coles)
This sys_clone() call is basically fork() - the child process is
mostly identical to the parent process, including having all the same
address space mappings and not having any free virtual address space as a
result. The context is still compromised, because it's the same context.
The only way to be running in a safe environment in the crash
handler is to have the dump generation run inside another process that was
already running before the crash, or to exec() another binary after forking
to actually get a fresh environment, and breakpad cannot do either of those
things.
OOM crashes on 32-bit android aren't very interesting in general -
address space is very limited and it's quite possible to run out without
anything actually being wrong; some devices also have bad graphics drivers
that leak address space. It's unlikely you can do anything to address these
crashes even if you had a dump.

Thanks for your answer.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context.

Post by Torne (Richard Coles)

It is very likely that the problem is derived from an OOM crash.
From the source code, we can see when crashed the crashed

process call system function 'clone' to create a child-process which is
responsible for minidump.

Post by Torne (Richard Coles)

Under this mode (not full out-of-process), the situation "runs

inside the crashed process in a compromised context" will also exist ?
=========================================================================
bool ExceptionHandler::GenerateDump(CrashContext *context) {
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg,
NULL, NULL,
NULL);
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);
SendContinueSignalToChild();
int status;
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));
sys_close(fdes[1]);
}
=========================================================================
On Wednesday, July 18, 2018 at 5:18:15 AM UTC+8, Torne (Richard

Post by Torne (Richard Coles)
On Android the process is usually 32-bit, which means address
space is very limited. When the address space is exhausted then you'll get
an OOM crash - and if it's bad enough, breakpad will also fail to function
correctly because it will not be able to find any free address space to
mmap() the things it needs to look at to generate a proper crash report.
If that's what's happening then there's no way to fix this - it's
a limitation of breakpad's design, which runs inside the crashed process in
a compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Version : 63.0.3239.111.

Post by Torne (Richard Coles)

--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Torne (Richard Coles)]

Correct.

Post by Liu Xiao
Thank you very much.
In our official builds it's impossible to use libunwind, because official
builds don't include unwind tables in the binary on-device, to reduce the
size of the binary.

Post by Torne (Richard Coles)

Could I understand like this : the minidump generated by breakpad is

the only method for analyzing crash on official builds, though there is the
defect "The minidump can not always generated" ?
On Wednesday, July 25, 2018 at 11:12:54 PM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)

Look forward to crashpad release.
Our team is now mainly using "unwind the stack (using conventional
libunwind)" for analyzing crash, and it is rarely failed.
Do you think it is a safe method for developer?

In our official builds it's impossible to use libunwind, because official
builds don't include unwind tables in the binary on-device, to reduce the
size of the binary.
If you're including the unwind tables then libunwind will probably
generally work but I have no idea whether that's actually
expected/guaranteed to be safe to call in a compromised context.
On Tuesday, July 24, 2018 at 11:01:44 PM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
Crashpad for android is currently being worked on and hopefully will be
enabled in the near future. I don't know about the status on Linux.
No crash handling solution originating from inside the app can
guarantee that a dump will always be generated perfectly, but crashpad runs
in a non-compromised context which makes the current most common failure
modes of breakpad impossible.

Post by Liu Xiao
Thank you very much.
I want to ask a question that when crashpad can be implemented on
Linux and Android ? And it is sure to resolve the problem "The minidump can
not always generated by breakpad" ?

Post by Torne (Richard Coles)

Post by Liu Xiao
Thank you very much.
his sys_clone() call is basically fork() - the child process is
mostly identical to the parent process.

Post by Torne (Richard Coles)

Yes, creating child-process by sys_clone() is not the full

out-of-process mode.

I'm not sure what you're referring to here.

Post by Liu Xiao
The only way to be running in a safe environment in the crash
handler is to have the dump generation run inside another process.

Post by Torne (Richard Coles)

Yes, I have got it from the breakpad doc, that is called

out-of-process.

I'm not aware of breakpad having any other way to operate - we're
replacing it with crashpad in part to avoid this limitation.

Post by Liu Xiao

Post by Torne (Richard Coles)

I want to implement it before, but it is sensitive for creating

a new process, and I am not sure if it will resolve the problem in OOM
Crash, what do you think ?

I'm not sure what you're asking; as I said we don't use/support any
other way to handle crashes on android right now (as we haven't yet moved
to crashpad) and I'm not aware of breakpad having any alternative.

Post by Liu Xiao
OOM crashes on 32-bit android aren't very interesting in general

Post by Torne (Richard Coles)

Do you mean OOM Crash is a bothersome problem on 32-bit Android?

No, I mean that it's not generally considered important to
investigate OOM crashes because they very rarely lead to finding an actual
bug that can be fixed - so we don't consider it a huge problem that
breakpad fails to generate a dump in some OOM situations. (though it is
still a bit of a problem, and we're moving to crashpad in part to avoid it).

Post by Liu Xiao

Post by Torne (Richard Coles)

I am using Android 7.0, it is definitely a 32-bit Android, or

there may be a 64-bit version ? How to comfirm it ?

It's not about what version of Android you are running - what matters
is whether the chromium build you are using is 32-bit or 64-bit. By default
it's 32-bit, and while it's possible to build a 64-bit build for some
configurations (though not all; monochrome is always 32-bit for example) we
don't ship the 64-bit version to anyone and generally don't recommend that
people use it, because it currently uses quite a lot more memory, to the
point that the user experience is significantly worse. Unless you
explicitly chose to build the 64-bit version when configuring your build,
you are using the 32-bit version.

Post by Liu Xiao
On Thursday, July 19, 2018 at 12:04:22 AM UTC+8, Torne (Richard

Post by Torne (Richard Coles)
This sys_clone() call is basically fork() - the child process is
mostly identical to the parent process, including having all the same
address space mappings and not having any free virtual address space as a
result. The context is still compromised, because it's the same context.
The only way to be running in a safe environment in the crash
handler is to have the dump generation run inside another process that was
already running before the crash, or to exec() another binary after forking
to actually get a fresh environment, and breakpad cannot do either of those
things.
OOM crashes on 32-bit android aren't very interesting in general -
address space is very limited and it's quite possible to run out without
anything actually being wrong; some devices also have bad graphics drivers
that leak address space. It's unlikely you can do anything to address these
crashes even if you had a dump.

Thanks for your answer.
If that's what's happening then there's no way to fix this - it's
a limitation of breakpad's design, which runs inside the crashed process in
a compromised context.

Post by Torne (Richard Coles)

It is very likely that the problem is derived from an OOM crash.
From the source code, we can see when crashed the crashed

process call system function 'clone' to create a child-process which is
responsible for minidump.

Post by Torne (Richard Coles)

Under this mode (not full out-of-process), the situation "runs

inside the crashed process in a compromised context" will also exist ?
=========================================================================
bool ExceptionHandler::GenerateDump(CrashContext *context) {
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg,
NULL, NULL,
NULL);
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);
SendContinueSignalToChild();
int status;
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));
sys_close(fdes[1]);
}
=========================================================================
On Wednesday, July 18, 2018 at 5:18:15 AM UTC+8, Torne (Richard

Post by Torne (Richard Coles)
On Android the process is usually 32-bit, which means address
space is very limited. When the address space is exhausted then you'll get
an OOM crash - and if it's bad enough, breakpad will also fail to function
correctly because it will not be able to find any free address space to
mmap() the things it needs to look at to generate a proper crash report.
If that's what's happening then there's no way to fix this - it's
a limitation of breakpad's design, which runs inside the crashed process in
a compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Version : 63.0.3239.111.

Post by Torne (Richard Coles)

--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Liu Xiao]

though not all; monochrome is always 32-bit for example

Post by Torne (Richard Coles)

Recently I have seen "monochrome " many times, is it a lite version of

chrome ?

Post by Torne (Richard Coles)

I can't get the precise definition of it, could you help me?
Thank you very much.
his sys_clone() call is basically fork() - the child process is mostly
identical to the parent process.

Post by Torne (Richard Coles)

Yes, creating child-process by sys_clone() is not the full

out-of-process mode.

I'm not sure what you're referring to here.

The only way to be running in a safe environment in the crash handler is
to have the dump generation run inside another process.

Post by Torne (Richard Coles)

Yes, I have got it from the breakpad doc, that is called

out-of-process.

I'm not aware of breakpad having any other way to operate - we're
replacing it with crashpad in part to avoid this limitation.

Post by Torne (Richard Coles)

I want to implement it before, but it is sensitive for creating a new

process, and I am not sure if it will resolve the problem in OOM Crash,
what do you think ?

I'm not sure what you're asking; as I said we don't use/support any other
way to handle crashes on android right now (as we haven't yet moved to
crashpad) and I'm not aware of breakpad having any alternative.

OOM crashes on 32-bit android aren't very interesting in general

Post by Torne (Richard Coles)

Do you mean OOM Crash is a bothersome problem on 32-bit Android?

No, I mean that it's not generally considered important to investigate OOM
crashes because they very rarely lead to finding an actual bug that can be
fixed - so we don't consider it a huge problem that breakpad fails to
generate a dump in some OOM situations. (though it is still a bit of a
problem, and we're moving to crashpad in part to avoid it).

Post by Torne (Richard Coles)

I am using Android 7.0, it is definitely a 32-bit Android, or there

may be a 64-bit version ? How to comfirm it ?

It's not about what version of Android you are running - what matters is
whether the chromium build you are using is 32-bit or 64-bit. By default
it's 32-bit, and while it's possible to build a 64-bit build for some
configurations (though not all; monochrome is always 32-bit for example) we
don't ship the 64-bit version to anyone and generally don't recommend that
people use it, because it currently uses quite a lot more memory, to the
point that the user experience is significantly worse. Unless you
explicitly chose to build the 64-bit version when configuring your build,
you are using the 32-bit version.

On Thursday, July 19, 2018 at 12:04:22 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
This sys_clone() call is basically fork() - the child process is mostly
identical to the parent process, including having all the same address
space mappings and not having any free virtual address space as a result.
The context is still compromised, because it's the same context.
The only way to be running in a safe environment in the crash handler is
to have the dump generation run inside another process that was already
running before the crash, or to exec() another binary after forking to
actually get a fresh environment, and breakpad cannot do either of those
things.
OOM crashes on 32-bit android aren't very interesting in general -
address space is very limited and it's quite possible to run out without
anything actually being wrong; some devices also have bad graphics drivers
that leak address space. It's unlikely you can do anything to address these
crashes even if you had a dump.

Thanks for your answer.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context.

Post by Torne (Richard Coles)

It is very likely that the problem is derived from an OOM crash.
From the source code, we can see when crashed the crashed process

call system function 'clone' to create a child-process which is responsible
for minidump.

Post by Torne (Richard Coles)

Under this mode (not full out-of-process), the situation "runs

inside the crashed process in a compromised context" will also exist ?
=========================================================================
bool ExceptionHandler::GenerateDump(CrashContext *context) {
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg, NULL,
NULL,
NULL);
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);
SendContinueSignalToChild();
int status;
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));
sys_close(fdes[1]);
}
=========================================================================
On Wednesday, July 18, 2018 at 5:18:15 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
On Android the process is usually 32-bit, which means address space is
very limited. When the address space is exhausted then you'll get an OOM
crash - and if it's bad enough, breakpad will also fail to function
correctly because it will not be able to find any free address space to
mmap() the things it needs to look at to generate a proper crash report.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Version : 63.0.3239.111.

Post by Torne (Richard Coles)

--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss

[Torne (Richard Coles)]

Post by Torne (Richard Coles)
though not all; monochrome is always 32-bit for example

Post by Torne (Richard Coles)

Recently I have seen "monochrome " many times, is it a lite version of

chrome ?

Post by Torne (Richard Coles)

I can't get the precise definition of it, could you help me?

Monochrome is a version of chrome for android that can also function as the
device's implementation of WebView, for use in other apps. Unless you are
building your own custom android OS image, Monochrome is of no use to you -
only apps specifically designated by the OS can be used as a webview
implementation.

Monochrome is significantly larger than regular chrome, because it has to
include both 32 and 64 bit versions of the webview code (it's just that
it's smaller than the *total* size of the separate versions of chrome and
webview, so it's a net saving when it replaces those).

Post by Torne (Richard Coles)

Post by Torne (Richard Coles)

Thank you very much.
his sys_clone() call is basically fork() - the child process is mostly
identical to the parent process.

Post by Torne (Richard Coles)

Yes, creating child-process by sys_clone() is not the full

out-of-process mode.

I'm not sure what you're referring to here.

The only way to be running in a safe environment in the crash handler is
to have the dump generation run inside another process.

Post by Torne (Richard Coles)

Yes, I have got it from the breakpad doc, that is called

out-of-process.

I'm not aware of breakpad having any other way to operate - we're
replacing it with crashpad in part to avoid this limitation.

Post by Torne (Richard Coles)

I want to implement it before, but it is sensitive for creating a

new process, and I am not sure if it will resolve the problem in OOM Crash,
what do you think ?

I'm not sure what you're asking; as I said we don't use/support any other
way to handle crashes on android right now (as we haven't yet moved to
crashpad) and I'm not aware of breakpad having any alternative.

OOM crashes on 32-bit android aren't very interesting in general

Post by Torne (Richard Coles)

Do you mean OOM Crash is a bothersome problem on 32-bit Android?

No, I mean that it's not generally considered important to investigate
OOM crashes because they very rarely lead to finding an actual bug that can
be fixed - so we don't consider it a huge problem that breakpad fails to
generate a dump in some OOM situations. (though it is still a bit of a
problem, and we're moving to crashpad in part to avoid it).

Post by Torne (Richard Coles)

I am using Android 7.0, it is definitely a 32-bit Android, or there

may be a 64-bit version ? How to comfirm it ?

It's not about what version of Android you are running - what matters is
whether the chromium build you are using is 32-bit or 64-bit. By default
it's 32-bit, and while it's possible to build a 64-bit build for some
configurations (though not all; monochrome is always 32-bit for example) we
don't ship the 64-bit version to anyone and generally don't recommend that
people use it, because it currently uses quite a lot more memory, to the
point that the user experience is significantly worse. Unless you
explicitly chose to build the 64-bit version when configuring your build,
you are using the 32-bit version.

On Thursday, July 19, 2018 at 12:04:22 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
This sys_clone() call is basically fork() - the child process is mostly
identical to the parent process, including having all the same address
space mappings and not having any free virtual address space as a result.
The context is still compromised, because it's the same context.
The only way to be running in a safe environment in the crash handler
is to have the dump generation run inside another process that was already
running before the crash, or to exec() another binary after forking to
actually get a fresh environment, and breakpad cannot do either of those
things.
OOM crashes on 32-bit android aren't very interesting in general -
address space is very limited and it's quite possible to run out without
anything actually being wrong; some devices also have bad graphics drivers
that leak address space. It's unlikely you can do anything to address these
crashes even if you had a dump.

Thanks for your answer.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context.

Post by Torne (Richard Coles)

It is very likely that the problem is derived from an OOM crash.
From the source code, we can see when crashed the crashed process

call system function 'clone' to create a child-process which is responsible
for minidump.

Post by Torne (Richard Coles)

Under this mode (not full out-of-process), the situation "runs

inside the crashed process in a compromised context" will also exist ?
=========================================================================
bool ExceptionHandler::GenerateDump(CrashContext *context) {
const pid_t child = sys_clone(
ThreadEntry, stack, CLONE_FS | CLONE_UNTRACED, &thread_arg,
NULL, NULL,
NULL);
// Close the read end of the pipe.
sys_close(fdes[0]);
// Allow the child to ptrace us
sys_prctl(PR_SET_PTRACER, child, 0, 0, 0);
SendContinueSignalToChild();
int status;
const int r = HANDLE_EINTR(sys_waitpid(child, &status, __WALL));
sys_close(fdes[1]);
}
=========================================================================
On Wednesday, July 18, 2018 at 5:18:15 AM UTC+8, Torne (Richard Coles)

Post by Torne (Richard Coles)
On Android the process is usually 32-bit, which means address space
is very limited. When the address space is exhausted then you'll get an OOM
crash - and if it's bad enough, breakpad will also fail to function
correctly because it will not be able to find any free address space to
mmap() the things it needs to look at to generate a proper crash report.
If that's what's happening then there's no way to fix this - it's a
limitation of breakpad's design, which runs inside the crashed process in a
compromised context. The future replacement crash handling system
(crashpad) works in a different way that doesn't have this problem but is
not yet ready for Android.

Version : 63.0.3239.111.

Post by Torne (Richard Coles)

--
--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss
--

--
http://groups.google.com/a/chromium.org/group/chromium-discuss